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Foreword 
Graham Smith 


Deliberative democracy is a contested field of study. Its central practice of deliber- 
ation has a number of generally accepted features. One of these is the celebration 
of pluralism. Pluralism has value in both normative and epistemological terms. 
Deliberation rests on the presence, articulation, and consideration of a plurality 
of perspectives. Inclusion of the voices of the full diversity of social groups is a 
moral foundation of deliberative democracy. Equally, pluralism underpins collec- 
tive judgement. Only through recognizing and understanding the perspectives of 
diverse others do we come to considered political judgements. Deliberative democ- 
racy requires openness and respect towards pluralism: to different ways of seeing the 
world and in the collective search for meaning. 

If the practice of deliberation rests on openness and respect towards different ways 
of seeing and meaning making, the same holds true for the study of deliberation. We 
require openness and respect towards different ways of seeing our object of analysis. 
Just as pluralism is a foundational value for deliberative practice, so it should also be 
for research on deliberation. 

Deliberative democracy is a political project that embraces conceptual analysis 
through practical action in varying forms. The questions we ask of that political 
project will vary. And the way we aim to answer those questions will vary too. Social 
science is at its worst when method comes before questions; when we dogmatically 
make sense of the world through only one methodological frame. 

My own engagement with deliberative democracy as a field of study has gener- 
ated a raft of questions, the answers to which have required the application of a 
range of methods. When troubled by the potential implications of the deliberative 
systems perspective undermining the core justification of deliberative democracy, 
tackling this question meant engaging in normative democratic theorizing. When 
questioning whether citizens’ assemblies could tackle contentious issues such as 
Brexit, colleagues and I not only adopted a positivist experimental design, but also 
had to organize a national four-day assembly! When making sense of why the inno- 
vative NHS Citizen participatory system had failed, an interpretivist framework was 
embraced, enabling us to investigate the perspectives and motivations of different 
actors towards the legitimacy of the initiative. In these and other research enterprises, 
I have had the pleasure of working with colleagues with very different methodological 
orientations who have forced me to reflect on my own methodological preconditions 
and prejudices and, I believe, this has led to better work in the process. The main 


vi Foreword 


lesson I have learned: we should not be afraid to venture across traditional lines of 
methodological contention. 

Just as deliberative democracy as a political project demands openness, respect, 
and listening across difference, the same needs to be true for our application of differ- 
ent methodologies. Entertaining different methodological orientations and research 
methods takes us out of our comfort zones, challenges our prejudices, and makes us 
see the world in different ways. These are virtues of deliberative processes—and they 
should also be the virtues of the research community that is engaged in their study. 

While this book is written primarily for researchers, I have a strong sense that it 
will prove valuable to the deliberative practitioner and activist communities as well. 
This is not simply because it provides important insights into how to evaluate and 
draw comparisons across discrete projects. Most practitioners I know have a broader 
set of concerns about how their work can be understood in more systemic terms: 
broader questions of citizen empowerment and civic renewal. The chapters in this 
collection may well inspire approaches to evaluation and reflection that respond to 
these systemic and political questions. 

The embrace of methodological pluralism that underpins this impressive collec- 
tion is the sign ofa mature and reflexive community of practice. That this book project 
is directed from the Centre for Deliberative Democracy and Global Governance at 
the University of Canberra should not be a surprise. This is a research centre that has 
promoted and sustained a pluralist orientation to the study of deliberative democ- 
racy. We should thank the Centre, editors, and contributors for modelling the kind of 
disposition towards methodology that is essential for good social science. Our object 
of study is a set of reflective practices. Such reflexivity in the application of research 
methods must define our collective project of sense-making. 
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1 
Researching Deliberative Democracy 
Methods and Approaches 


SelenA. Ercan, Hans Asenbaum, Nicole Curato, and Ricardo F. 
Mendonca 


As the field of deliberative democracy continues to grow, methodological questions 
loom large: Which methods are a good fit for the study of deliberative democracy? 
How can we translate normative theory into empirical research? Can the methods 
used for the study of deliberation in small-scale settings be used to assess deliberation 
at the large scale? 

This book aims to answer these and many other questions and showcase a variety 
of methods and approaches used in deliberative democracy research. We identify the 
strengths and limitations of each method and reflect on how different methods can 
be combined to generate a comprehensive and multidimensional account of delib- 
erative democracy. As editors of this book, our goal is both ambitious and modest. 
It is ambitious insofar as we hope to provide a ‘go-to resource’ for anyone wishing 
to study deliberative democracy. It is modest insofar as we recognize that this book, 
or any book for that matter, can never be complete in its coverage of methods, espe- 
cially in a dynamic and growing field of study. Nevertheless, we have put our best 
efforts into curating a volume that features both established and emerging methods 
for researching deliberative democracy. 

There are more than thirty methods covered in this book, approaching the study 
of deliberative democracy from different angles and engaging with different kinds of 
research questions and agendas. Some methods zero in on specific characteristics of 
deliberative practices, while others address broader questions in the field. By bring- 
ing all these diverse methods together, the book aims to practice what deliberative 
democracy preaches: enabling reflection and advancing critical engagement across 
different perspectives. We hope that our readers receive this book as an invitation to 
reflect on, evaluate, and articulate their assumptions about what deliberative democ- 
racy isand howit can be studied. At the same time, we hope that it encourages them to 
engage across different methods and approaches and contribute to the development 
of deliberative democracy as an innovative, reflexive, and inclusive field of study. 

We begin this chapter by providing a brief overview of deliberative democracy 
research and explaining how the normative theory interacts with empirical research 
in the field. This is followed by a discussion of how deliberative ideals shape the 
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practice and purpose of research. We show how scholars from different disciplinary 
and methodological backgrounds go about researching deliberative democracy and 
outline the key approaches and methods they employ. The chapter concludes with 
an overview of thirty-one different methods included in this volume. We hope that 
upon reading this book, or parts of it, readers will feel inspired to undertake research 
on deliberative democracy and thus advance the capacity of the field to address the 
problems facing contemporary democracies. 


Bridging Normative Theory and Empirical Research 


Deliberative democracy is a political ideal where ‘people come together on the basis 
of equal status and mutual respect, to discuss the political issues they face, and, on 
the basis of those discussions, decide on the policies that will then affect their lives’ 
(Bachtiger et al. 2019, 2). It is a normative theory about how collective decisions 
ought to be made (Habermas 1992), as well as a political project that advances prac- 
tical ways of moving contemporary societies in a deliberative direction (Curato et al. 
2018). Over the past two decades, we have witnessed various practical applications 
of deliberative democracy, from the ‘wave’ of deliberative mini-publics around the 
world to social movements’ deliberative decision-making and authoritarian regimes 
using deliberative processes to listen to the people (della Porta and Rucht 2013; He 
and Warren 2011; OECD 2020). The rise of deliberative practices in diverse settings 
and different countries offers a rich ground for researchers to conduct theoreti- 
cal and empirical research on deliberative democracy, while also raising significant 
methodological questions. 

One central methodological question in the field of deliberative democracy is how 
to bridge normative theory and empirical research. Normative theory is about the 
question of ‘what ought to be, whereas empirical research focuses on ‘what is actually 
happening. There were several key debates within the field, which sought to clar- 
ify the relationship between normative theory and empirical research (Mutz 2008; 
Neblo 2005; Thompson 2008). While sceptics may view the normative nature of 
deliberative democracy as a sign of its inapplicability to an imperfect world, empir- 
ical researchers have shown various ways in which normative theory interacts with 
real-world politics, and how this interaction can further both theory and practice. 

Some scholars use normative theory to develop hypotheses to be empirically tested 
(see, for example, Gronlund et al. 2010). They draw on deliberative ideals to create 
experimental settings to test the conditions required for successful deliberation and 
explore ways to improve deliberative quality. For example, they investigate whether 
the introduction of deliberative norms in like-minded discussions alleviates group 
polarization (Strandberg et al. 2019). Others conduct experiments in ‘real-world’ set- 
tings with ‘real-world’ actors, such as elected officials. Drawing on deliberative ideals, 
Michael Neblo and his colleagues (2018), for example, design deliberative town halls 
where citizens meet and interact over the Internet with their elected representatives. 
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In these studies, the normative theory of deliberative democracy serves as a key 
starting point informing experimental design, and the hypotheses to be tested in the 
course of these experiments. 

Other scholars use the normative ideals of deliberative democracy as a frame- 
work for interpreting or assessing the political dynamics in existing democracies 
(e.g. Fan 2020; Hendriks et al. 2020). Taking the ideal conditions for deliberation as 
their benchmark, they examine the deliberative quality of discussion in both small 
groups and the broader public sphere. They have, for example, investigated the 
large-scale processes of deliberation on various issues such as climate governance 
(Stevenson and Dryzek 2014), LGBT equality (Barvosa 2018), and animal rights 
(Parry 2017). Meanwhile, some scholars take an inductive approach, and use empiri- 
cal work to sharpen deliberative theory’s normative claims (Doerr 2018; Curato 2019; 
Asenbaum, Chapter 5 in this volume) or use empirical research to bring delibera- 
tive democracy’s principles to life (e.g. Cunningham and Tamale, Chapter 30 in this 
volume). 

These studies and many others show that normative theory and empirical research 
can be combined in ways that advance both. The field moves back and forth, between 
theory and empirics, resulting in their mutual enrichment. Empirical research can 
throw new light on the normative questions that deliberative democrats are grap- 
pling with. The insights gained from the close study of deliberative practices (in both 
real-life and experimental settings) help ‘in the process of identifying normative prin- 
ciples themselves’ (Bachtiger 2019, 657). Take, for example, Nicole Doerr’s (2018) 
ethnographic research on political translation. Doerr makes a case for the role of 
translators not as ‘neutral’ actors in a deliberative process but as disruptors, or the 
‘third voice’ for marginalized participants to be heard and understood. In this regard, 
Doerr’s work challenges deliberative theory’s assumptions about the virtue of neu- 
trality in facilitating deliberation, and instead emphasizes the need for challenging 
structures of inequality for deliberative goals to be realized. 

This approach to empirical research and theoretical refinement speaks to John 
Dryzek’s call to use empirical research to refine the theory and make it ‘more sen- 
sitive to real-world constraints and opportunities’ (Dryzek 2007, 240). In this sense, 
theoretical ideals should not be viewed as fixed but as contingent points of orientation 
that, informed by empirical experience, adapt, mature, and change over time. Empir- 
ical research enables us to capture what has not been theorized before and hence 
produce new ideals and contribute to the development of what Simone Chambers 
calls ‘critical applied theories’ (Chambers, Chapter 2 in this volume). 


Diversity of Research Methods 


The development and refinement of deliberative ideals is driven by a variety of theo- 
retical and empirical methods. The field draws on established methods such as survey 
research, field experiments, ethnography, or narrative analysis. At the same time, 
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it also creates new methods of its own. The Discourse Quality Index (DQI) is one 
of the best-known methods specifically designed to operationalize norms of delib- 
eration as conceptualized by Habermas's normative theory. It originally sought to 
provide a tool for the measurement and comparison of the deliberativeness of parlia- 
mentary debates (Bachtiger et al. 2005; Steiner et al. 2004), although it has also been 
applied to assess the quality of deliberative mini-publics amongst lay citizens (Him- 
melroos 2017). Other specific methods developed in the field include the Deliberative 
Reason Index (Niemeyer and Veri, Chapter 7 in this volume), the Listening Qual- 
ity Index (Scudder, Chapter 8 in this volume), and the Online Deliberative Matrix 
(Kies, Chapter 10 in this volume). These methods point to the relevance of constant 
innovation in deliberative democracy research. 

The scholarship on deliberative democracy is built on a diverse epistemic com- 
munity. This diversity is reflected in the methods included in this book. Scholars 
from various research traditions, disciplinary backgrounds, and geographic exper- 
tise present the methods they use while researching deliberative democracy. These 
include the methods used not only by empirically oriented scholars, but also those 
adopted by political theorists. While questions of methods are often thought to be 
something only empirical scholars should worry about, we show that theoretically 
oriented scholars also need methods (Chambers, Chapter 2 in this volume). The 
book offers a glimpse into the toolboxes of both theoretical and empirical scholars 
of deliberative democracy and presents the methods, and approaches they use for 
researching deliberative democracy. Theorists, political scientists, sociologists, pol- 
icy analysts, and communication scholars contribute to the volume by presenting 
a distinctive angle on how deliberative democracy can be researched and further 
improved. They showcase examples of deliberation happening not only in struc- 
tured forums or face-to-face settings, but also in the messy public sphere, in and 
through various media outlets, the Internet, social movements and everyday conver- 
sations. They offer a variety of methods and approaches that can be used to study not 
only textual and verbal communication but also nonverbal communication includ- 
ing visuals, colours, sound, silence, presence or absence in public deliberation (Ercan 
and Hendriks, Chapter 22 in this volume, Mendonga et al 2022 ) 

Some may find it perplexing that there are many methodological possibilities for 
researching deliberation, and yet no single approach is agreed to be the best. There 
are also different comprehensions of deliberative democracy. Diana Mutz (2008, 525) 
may have been exaggerating when she stated that ‘there are as many definitions 
of deliberation as there are theorists’; nevertheless, it is accurate to say that there 
has been an expansion of conceptual approaches and interpretations of delibera- 
tive democracy over the years (Bachtiger and Hangartner 2010). This raises concerns 
about ‘concept stretching, and a worry that normative theory will lose its prescriptive 
force (Goodin 2018, 883). In response to this danger, Robert Goodin suggests dis- 
tinguishing between different kinds of concept stretching, and accepting only those 
that are good for deliberative theory. In his view, we may stretch the original vision of 
deliberative democracy (developed by Habermas) for the purposes of making it ‘more 
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democratic’ (meaning more inclusive), ‘more deliberative’ (meaning open to rele- 
vant information, and better informed), or ‘more realistic’ (meaning more applicable 
to the real world). These three conditions can be read as the shared commitments 
of deliberative democracy scholarship. What scholars share is not a rigid definition 
of deliberation, but a set of commitments that drive research and progress in the 
field. 

Rather than searching for ‘the correct definition of deliberation, we could under- 
stand deliberation as contingent, dependent on different contexts and goals, as 
suggested by André Bachtiger and John Parkinson (2019). On this account, the con- 
ditions for ‘good deliberation’ change depending on whether we seek to realize them 
in formal institutions or the public sphere. Additionally, the ‘systemic turn’ brings 
new ways of characterizing and assessing deliberation by shifting the focus from 
structured forums to multiple other sites of deliberation (Elstub et al. 2019; Steiner 
et al. 2017). Here, the conditions for good deliberation are defined more dynami- 
cally. In a healthy deliberative system, good deliberation is not necessarily evenly 
distributed; the low quality of deliberation in certain sites is compensated for by high- 
quality deliberation in other sites (Dryzek 2009). More importantly, low deliberative 
quality may accompany or even be an integral part of protests or other political orga- 
nizations that add information and draw attention to issues in the deliberative system 
(Parry 2017). These suggestions open new ways of understanding and studying 
deliberation. 

Despite the dynamism and progress of deliberative scholarship in the past decades, 
there is still a lot to do. More needs to be done, especially in sharpening the crit- 
ical edge of our research agendas and broadening our sources of knowledge. For 
example, we, as editors of this book, recognize that we need to pay attention to the 
process of knowledge production, and work towards making it more inclusive and 
democratic (see Asenbaum, Chapter 5 in this volume). Scholars of critical race stud- 
ies and feminist researchers have long criticized social science methods for using 
the logic of extraction where researchers ‘take, hit, and run’ (Reinharz 1992, 95). 
Deliberative democracy research is not immune from these critiques, for many of 
the methods we use are legacies of colonial traditions which we replicate as we 
study different societies (Banerjee 2021; Moran and Ross 2021; also see Smith 1999). 
While most researchers do follow guidelines for ethical conduct in ensuring the 
safety and dignity of participants, we understand the wider demand that our research 
be a non-exploitative process that views people as participants in co-producing 
knowledge. 

We also witness the emergence of movements to decolonize knowledge in differ- 
ent disciplines. Indeed, deliberative scholarship needs to re-examine its assumptions 
about the state, citizenship, and even its own core normative principles as hinged on a 
particularistic history of Western democracies. Finally, we recognize the domination 
of countries in the Global North as centres of knowledge production, although we 
now see increasing recognition of different centres of knowledge production in the 
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field of deliberative democracy. Despite these and many other open questions, there 
is considerable scope for challenging deliberative democracy to continue evolving 
and realizing its emancipatory promise. 


Practicing What We Preach: The Purpose of Deliberative 
Democracy Research 


The scholarship on deliberative democracy is interested in thinking about ways to 
improve democratic practice in various settings, ranging from parliamentary debates 
to everyday conversations. The normative ideals of deliberative democracy not only 
provide an analytical framework that inform the generation and analysis of data, but 
also shape the purpose and process of research in important ways. 

Just as feminist ideals shape the purpose and conduct of feminist research, for 
example by requiring researchers to reflect on the significance of gender and gender 
asymmetry in their work, and utilize their work for advancing women’s empow- 
erment (Fonow and Cook 2005), deliberative ideals such as inclusion, diversity, 
listening, or openness to new ideas shape the conduct and purpose of research 
in deliberative democracy. Deliberative democrats aim to problematize exclusions 
and marginalization or illegitimate decision-making in a democracy and create 
the conditions for meaningful political communication. They aim to fulfil the 
democratic ideal of emancipation by disturbing existing democratic practices and 
prioritizing inclusive forms of engagement (Wojciechowska 2019). In this sense, 
the tradition of critical theory, in which the goal of knowledge production is 
emancipation from domination (Hammond 2019; Mendonça 2013), is essential to 
deliberative democracy as a field, even though not all deliberative democrats see 
themselves as critical theorists. Critical theory, which originally emerged in the 
Institute for Social Research (or the ‘Frankfurt School’) in Germany in the 1920s 
and 1930s, refers to a particular mode of research and analysis. For critical the- 
orists, the purpose of scientific inquiry is to serve human as well as nonhuman 
interests and illuminate topics that ordinary people care about (Smith 2002). This 
requires putting philosophical questions under the spotlight of empirical social sci- 
ence research, with the aim being to effect change in society and enable freedom from 
oppression. 

Some scholars are more explicit than others about these normative commitments 
and their influence on the topics they choose to study, or the type of research they 
undertake. They explicate how deliberative norms shape their research and view this 
process of explanation as strengthening their research and its validity. In this volume, 
Genevieve Fuji Johnson argues that research informed by deliberative democratic 
theory cannot bea value-free exercise. The ethically appropriate role ofa researcher is 
to stand in solidarity with groups, communities, and nations experiencing oppression 
and seeking justice as they define it. ‘Solidaristic research involves recognizing the 
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privilege and power we have as scholars and deploying our resources of social capital, 
time, and mode toward ending forms of expressior (Johnson 2021, 15). Johnson 
invites scholars of deliberative democracy to conduct solidaristic research, and take 
a stance in solidarity with, and empower particularly, those seeking racial justice, 
such as Indigenous peoples. We should do so, she argues, not simply to feel good 
about ourselves and our efforts but ‘to contribute constructively and purposefully 
towards their liberation and resurgence’ (Johnson, Chapter 4 in this volume). In her 
view, this is how researchers can help to fulfil the emancipatory ideals of deliberative 
democracy. 

Not everybody might agree with this level of normative involvement on the part 
of researchers in the research process. Researchers can play different roles in delib- 
erative democracy research (Evans and Kotchetkova 2009). Sometimes their role 
is to design deliberative processes, other times their role involves evaluating these 
processes. Nevertheless, the attempt to practice what the field preaches remains a 
frequent feature of deliberative democracy research. Researchers, drawing on delib- 
erative ideals, have engaged with pressing controversies, such as the abortion debate 
in Ireland, and established deliberative institutions to break the years of political 
deadlock facing this issue (Farrell and Suiter 2019). The increasing popularity of cit- 
izens’ assemblies in Belgium, France, Ireland, and the United Kingdom speaks to a 
similar practical agenda. Deliberative democracy researchers are a key driver of this 
development; they advocate institutional reform that enables citizens to deliberate 
on political issues in an inclusive, reasoned, and consequential manner. (e.g. Curato 
et al. 2021; Farrell and Suiter 2019; Renwick et al. 2018). 

Finally, some researchers draw on deliberative ideals within their own research 
and make research processes more deliberative. They do so by including them- 
selves in a deliberative dialogue with their research participants. Ricardo Mendonca 
(2009), for example, uses this kind of dialogue in his engagement with the dwellers 
of former leprosy colonies in Brazil. These dialogues enable him to unpack how peo- 
ple with leprosy articulate their suffering and grievances in everyday conversations 
and in interaction with each other. When informed by the principles of delibera- 
tive democracy, qualitative interviews can generate what Nicole Curato (2012) calls 
‘intersubjective knowledge. Such form of knowledge is developed ‘through a linguis- 
tic process of exchanging standpoints and bringing together different perspectives 
into a shared frame of understanding’ (Curato 2012, 577). Kei Nishiyama (2018) fol- 
lows a similar path and seeks this kind of knowledge production, when he conducts 
group dialogue to unpack the lived experiences of high school students in Japan. 
Epistemological diversity, justification, and reflection are essential dispositions to 
research practice for these scholars and many others. Marit Hammond describes this 
approach as ‘activist deliberative democracy, where the goal of researchers is not to 
provide solutions for laypeople but to treat them as ‘capable agents’ who diagnose 
their political problems and intervene to improve their situations (Hammond 2019, 
801). The idea is that the research is done with, rather than on, participants 
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(Bussu et al. 2020). Such activist or participatory research approaches put the polit- 
ical project of deliberative democracy at the centre of creating knowledge about 
deliberative democracy. 


Tackling Big Questions through Multiple Methods 


Many questions drive deliberative democracy research as it continues to grow in new 
directions. It would be impossible to do justice to the rich variety of questions that 
the field raises in this introduction. Yet, based on the chapters included in this book, 
we have identified four sets of questions that drive research in the field and four 
corresponding research approaches: theorizing, measuring, exploring, and enacting 
deliberation (see Table 1.1). 

The structure of the book reflects these four approaches in the study of deliberative 
democracy. These are, however, non-exhaustive categories. First, there are impor- 
tant and sometimes inevitable overlaps between the categories. Second, research in 
deliberative democracy entails many other activities, such as interpreting, evaluating, 
criticizing, and prescribing. Our four categories aim only to provide a heuris- 
tic to organize the multiplicity of research approaches and methods included in 
this book. 

Theorizing is a significant part of deliberative democracy research. It takes var- 
ious forms, from normative or critical theory to explanatory theory. Despite its 
significance, there is little guidance on how theorizing is done. Methods courses 
in political science, sociology or communication studies mostly involve training in 
empirical methods for data gathering, processing and analysis. They also involve 
exposure to key theoretical texts, but there is little, if any, guidance as to how the- 
orizing is done. They seem to assume that if students read theory, they will learn to 
write theory. In the social science literature, the few accounts that instruct how to 
theorize (Leopold and Stears 2008; Vincent 2004) are almost overwhelmed by the 
plethora of instruction on empirical methods. So how does theorizing work? What 
concrete steps does a theorist take in her research on deliberative democracy? How 


Table 1.1 Questions and Approaches in Deliberative Democracy Research 


Questions Approaches 


(1) What are the normative underpinnings and Theorizing deliberation 
implications of deliberative democracy? 

(2) What counts as good deliberation and what are Measuring deliberation 
its facets? 

(3) How is deliberation experienced in ‘real life’? Exploring deliberation 

(4) How can deliberative democracy be brought to Enacting deliberation 
action through research? 
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do theorists develop deliberative norms? This book presents concrete answers about 
the forms philosophical and theoretical engagement can take and how to employ 
empirical methods to develop deliberative theory (see Ackerly et al. 2021). As Jane 
Mansbridge (Chapter 33 in this volume) suggests, empirical researchers and prac- 
titioners can become theorists and theorists can become empirical researchers and 
practitioners. 

Measuring allows researchers to develop and operationalize the indicators for good 
deliberation based on a normative framework. The normative elaboration of the 
moral principles of deliberation and their translation into empirical indicators needs 
mathematical abilities, logical rigour, and contextual sensibility as much as theoret- 
ical acuity. Measuring deliberation is an important empirical aspiration with several 
clear gains. First, it allows for research on a large number of cases, and the conse- 
quent capacity to reach generalizable findings. Second, measuring deliberation helps 
establish causal explanations. Third, measurement allows for comparability amongst 
cases, facilitating clear conclusions. Hence, measuring deliberation can help tell us 
what does and does not work in the real world. It is essential to building more effective 
deliberative practices and systems. 

Exploring deliberation involves different ways of employing deliberative lenses to 
critically analyse the complexities and nuances of existing phenomena. Interpretive 
research methods are important in this regard (Ercan et al. 2017). Their capacity for 
nuanced engagement with a smaller number of cases allows researchers to lay bare 
the dynamics of deliberative exchanges. These methods enable researchers to stum- 
ble upon novelty and uncover the unexpected. Although deliberative theory works as 
a guiding framework, deliberative norms are not fixed a priori. Rather, they inspire 
empirical researchers to explore how deliberation works in the real world and how 
deliberative ideals might be expanded or adjusted. Rather than accepting as given a 
pre-established, external reality, the process of exploring deliberation co-constructs 
deliberative realities through the engagement of researchers and research partici- 
pants. Exploratory approaches also allow scholars to employ deliberative lens(es) 
to read and interpret a wide range of phenomena, such as the crisis of democ- 
racy, enabling them to redefine the problems and find possible solutions from a 
deliberative perspective (Hendriks et al. 2020). 

Finally, enacting deliberation in the context of research processes reconceptual- 
izes research as a democratic activity. Realizing deliberation in the research process 
itself can include qualitative, quantitative, and mixed methods. At the heart of enact- 
ing deliberation is a new understanding of the role of researchers and research 
participants. Contemporary science increasingly acknowledges the social respon- 
sibility of researchers and makes central the real-world relevance of research. Yet, 
understanding researchers to have a mandate and obligation for realizing social and 
political norms clashes with traditional understandings of researchers as detached, 
objective observers. Inspired by deliberative democratic norms and the tradition of 
action research, some deliberative scholars employ participatory methods that bring 
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participants as active agents into the research process. Rather than being sources 
of knowledge to be extracted, participants co-create knowledge on an equal footing 
with academic researchers (Bergold and Thomas 2012; Bussu et al. 2020). In doing 
so, they contribute to the process of democratizing knowledge production, as noted 
earlier. 

We have divided the thirty-one methods included in this book into these four 
approaches of researching deliberation. Table 1.2 provides an overview of these 
approaches and the associated methods. 


Navigating this Book 


The methods presented in the book are explained directly by those who are using 
them in their theoretical and/or empirical research on deliberative democracy in a 
variety of settings, contexts, and countries. Each chapter presents one method, elab- 
orates on its application in deliberative democracy, and offers illustrative examples 
showing how the method is used, or can be used, in practice. The chapters discuss 
the strengths of each method as well as its limitations. This kind of critical reflection 
comports with the deliberative values we embrace in the book, and it is crucial for 
the continuation of methodological development in the field. 


Part |: Theorizing Deliberation 


The chapters in Part I focus on theoretical investigations in the field. They emphasize 
different pathways to constructing theories of deliberative democracy and present 
theorizing as an important methodological skillset one can learn. Promoting an open 
and broad discussion on how to conduct theoretical research is essential for the 
development of the field and its agendas. 

Chapter 2 elaborates on different Methods of Theorizing in deliberative democ- 
racy research. Simone Chambers argues that while empirical social sciences have 
a relatively well-defined set of methods, tools, and approaches to work with when 
designing and undertaking research, political theorists have no clear toolbox to draw 
on. She introduces a typology of five types of normative theory: ideal theory, critical 
reflective theory, constructive reflective theory, critical applied theory, and construc- 
tive applied theory, and then demonstrates how each is developed and utilized within 
the deliberative democracy tradition. 

Chapter 3 argues that Formal Models are essential to promote conceptual clarity, 
which is necessary for empirical research. James Johnson argues that models should 
not be employed exclusively in ‘positive’ research. He notes the need to overcome 
a misguided dichotomy between positive and normative research, which artificially 
disentangles facts and values. Models are tools for the interpretation of reality and 


Table 1.2 Research Methods in Deliberative Democracy 


Theorizing Measuring Exploring Enacting 
Methods of Theorizing Discourse Quality Index Ethnography Deliberative Policy Analysis 
Formal Models Deliberative Reason Index Rhetorical Criticism Action Research 


Grounded Normative Theory 


Democratic Theorizing 


Listening Quality Index 


Macro-level Assessment of Deliberative 
Quality 

Online Deliberation Matrix 

Experimental Methods 

Deliberative Field Experiments 

Scenario Experiments 

Survey Methods 

Social Network Analysis 

Big Data Analysis 

Qualitative Comparative Analysis 


Process Tracing 


Q Methodology 
Dramaturgical Analysis 
Narrative Analysis 
Frame Analysis 
Talk-based Analysis 
Media Analysis 

Mixed Methods 

Case Study Research 


Community of Inquiry 


Deliberative Camp 


Source: Authors’ own table 


12 Researching Deliberative Democracy 


also lie at the heart of theoretical investigations. He illustrates the implications of 
theoretical modelling in the context of empirical research by focusing on the concept 
of ‘agreement’ used by many deliberative democrats. 

Chapter 4 focuses on the Grounded Normative Theory (GNT) approach in delib- 
erative democracy. Genevieve Fuji Johnson outlines the basic contours of GNT 
as a broad field and presents its solidaristic expression as a mode of inquiry 
that is capable of fostering justice and of expressing solidarity with oppressed 
groups. GNT blends empirical study with normative theorizing recursively, inclu- 
sively, accountably, and solidaristically. Johnson challenges deliberative democrats 
to develop their ‘critical muscles’ and to ensure that deliberative democracy remains 
relevant to addressing pressing political issues, including racialized injustice and 
oppression. 

Chapter 5 discusses Democratic Theorizing as a participatory approach to devel- 
oping democratic theory. Hans Asenbaum develops this approach by drawing on 
deliberative values of inclusion, diversity, listening, and transparency. In contrast 
to established approaches to theorizing democracy, it includes human and nonhu- 
man research participants in the theorizing process. Bringing together insights from 
grounded theory, participatory research, and assemblage theory, democratic the- 
orizing enhances the formative agency of those outside academia. Drawing on a 
democratic theorizing project with the Black Lives Matter movement, the chapter 
provides a step-by-step guide to take the reader through the different phases a 
theorizing project might take. 


Part Il: Measuring Deliberation 


Chapters in Part II present different ways of measuring the quality of deliberation 
in diverse settings, ranging from parliamentary debates to online discussion forums. 
Some chapters show how researchers can use the established social science methods, 
such as surveys, field experiments, or social network analysis, to examine delib- 
erative processes at different levels of political interaction. Other chapters present 
new methods, which are developed particularly for examining deliberative practices. 
These include the Discursive Quality Index, the Deliberative Reason Index, and the 
Listening Quality Index. 

Chapter 6 turns to the Discourse Quality Index (DQI) and provides guidance for 
those interested in adopting it in deliberative democracy research. André Bachtiger, 
Marlène Gerber, and Eléonore Fournier-Tombs survey the development of this 
method from its original to its expanded versions, including external and perception- 
based measurements. The authors also respond to some of the common criticisms 
of the DQI and reflect on novel developments in the automated measurement of 
deliberative quality. 
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Chapter 7 presents the Deliberative Reason Index, designed to assess how individ- 
uals reason together in a deliberative process. Simon Niemeyer and Francesco Veri 
introduce this method to capture the extent that a group coheres towards a shared 
understanding of the issue and its relevant dimensions. This index maps the inter- 
subjective consistency between actors to understand when and how deliberation 
improves reasoning. The chapter elaborates on the theoretical underpinning of the 
approach and the methods used to collect and analyse the results. 

Chapter 8 introduces the Listening Quality Index (LQI), an instrument that shifts 
the attention from speaking to listening in small-scale communicative interactions. 
Mary FE Scudder offers a critical review of the existing ways to measure listening in 
deliberation and highlights some of their limitations. She argues that some of these 
efforts go too far and equate listening with its ‘outcomes, while others do not go far 
enough and conflate listening simply with ‘the opportunity to hear. A less common 
approach to listening is to look for effects of listening on the ‘speaker’ instead of on 
the ‘listener. The LQI incorporates speaker satisfaction into a measure of listening 
and offers a lexical scale to measure the quality of listening in deliberative processes. 
The chapter outlines the type of data required for analysis, and how researchers can 
generate this data during and after the deliberative encounter. 

Chapter 9 presents the Macro-level Assessment of Deliberative Quality. Dannica 
Fleuß outlines a strategy for upscaling the measurement of deliberation to the 
nation-state level by combining elements from two strands of research: the method- 
ological standards of democracy measurements and the conceptual groundwork 
of systemic approaches to deliberation. Based on a review of previous measure- 
ment approaches, the chapter provides practical advice for the conceptualization, 
operationalization, and aggregation of procedures that allow valid measurement of 
nation-states’ deliberativeness. By drawing on this method, researchers can compare 
the deliberativeness of different democracies and identify the type of institutional 
reforms required to facilitate and promote deliberative democracy at the national 
level. 

Chapter 10 presents the Online Deliberative Matrix (ODM) as a method of mea- 
suring the quality of online deliberation. The method, introduced by Raphaél Kies, 
facilitates the assessment of political debates online through a matrix that measures 
three sets of criteria: the presence of deliberation, the deliberative attitudes of partici- 
pants, and the outcome of the deliberative process. The ODM can be applied to assess 
the deliberative quality of public debates taking place in the digital public sphere, 
including online forums designed for deliberation and common social media or news 
websites. The chapter also reflects on the utility of this method in light of the systemic 
turn in deliberative democracy and offers a critical review of attempts to scale up the 
analysis through automated assessment of online deliberation. 

Chapter 11 introduces Experimental Methods, detailing the kinds of experiments 
currently used in deliberative democracy research, including face-to-face, online, 
laboratory, and field experiments. Kimmo Grönlund and Kaisa Herne elaborate on 
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how they use experimental methods to examine and detect causal relationships in 
deliberative mini-publics, such as citizens’ assemblies, citizens’ juries, and deliber- 
ative polls. The chapter also provides insight into an experiment on enclave delib- 
eration and group polarization in deliberative processes. The authors reflect on the 
future of experimentation on deliberation and discuss the kinds of methodological 
innovations needed to advance the empirical research in the field. 

Chapter 12 focuses on Deliberative Field Experiments as a method for conducting 
experiments in real politics. Jon Kingzette and Michael Neblo define field experi- 
ments as systematic attempts to understand the causal dynamics of deliberation by 
manipulating features of the system: in naturalistic settings (rather than via surveys or 
in labs); on real political issues (rather than hypothetical scenarios); and by engaging 
a broad cross-section of people in a specific political jurisdiction potentially affected 
by pending political actions. The chapter outlines the process of deliberative field 
experiments and illustrates how they can be used to analyse deliberative events, such 
as the online town halls that Neblo and his colleagues have been organizing with the 
members of Congress in the US. 

Chapter 13 presents Scenario Experiments. Hannah Werner and Lala Muradova 
explain how they use this method to understand the impacts of deliberation on public 
opinion formation, democratic legitimacy, and political behaviour. They argue that 
scenario experiments are most useful when studying the micro mechanisms of inter- 
nal deliberation and the macro effects of deliberative events on the wider public. They 
show the application of this method in practice by providing examples from several 
recent research studies that use scenario experiments to analyze deliberation. The 
chapter also discusses how the methodological innovations in experimental social 
science research can improve research on deliberation. 

Chapter 14 discuses the many uses of Survey Methods in studying deliberation. 
John Gastil begins with a typology of survey methods to highlight the strengths 
and weaknesses of mail, phone, Internet, cross-sectional, and longitudinal surveys. 
The chapter then offers examples from the use of surveys of participants in singu- 
lar events, such as Deliberative Polls, the Irish Constitutional Convention, and the 
Australian Citizens’ Parliament, surveys of participants in laboratory experiments, 
and surveys of larger populations—often linked with deliberative events, such as 
the British Columbia Citizens’ Assembly or the Oregon Citizens’ Initiative Review. 
The chapter concludes by highlighting the ongoing problems in survey methods and 
reflects on the type of survey methods needed to span different levels of analysis to 
inform systemic theories of deliberation. 

Chapter 15 focuses on Social Network Analysis (SNA), exploring its potential for 
the study of deliberative processes and, particularly, deliberative systems. Eduardo 
Silva, Antônio Ribeiro, and Silvio Higgins show how SNA can be employed to explain 
if and how components of a system relate to each other, as well as different patholo- 
gies that may hinder the flow of ideas and proposals from one site to another. The 
chapter presents different examples illustrating the practical application of SNA 
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in deliberative democracy research and introduces the key metrics used by these 
investigations. 

Chapter 16 explores the use of Big Data Analysis in deliberative scholarship. Delv- 
ing into this innovative methodological trend, Núria Franco-Guillén, Sebastian De 
Laile, and John Parkinson define big data and set out methodological decisions 
necessary in big data analysis. The chapter emphasizes the change in scope repre- 
sented by the approach, which allows the analysis of massive volumes of data. The 
authors illustrate the merits and limits of big data analysis in deliberative democ- 
racy research by focusing on two case studies: the Scottish independence referendum 
campaign of 2012-2014, and the Australian campaign to recognize First Nations in 
its constitution. 

Chapter 17 elaborates on Qualitative Comparative Analysis (QCA) and shows how 
this method can be used for comparing different instances of deliberative processes. 
Matt Ryan explains how QCA combines the generalizability of quantitative research 
with the deep understanding of interpretive research. Through qualitative investi- 
gation of a small to medium number of cases, it identifies patterns that allow for 
valid conclusions. In deliberative democracy research, QCA is particularly useful for 
comparing a number of cases and understanding the conditions for good and bad 
deliberation. The chapter provides step-by-step guidance to undertaking QCA and 
demonstrates how lessons from QCA can be employed to design more successful 
deliberative forums in the future. 


Part Ill: Exploring Deliberation 


Chapters in Part III of the volume focus on the methods that are used to explore 
deliberation in structured forums and the public sphere. As readers will notice, some 
methods use similar data-gathering techniques but diverge in terms of the type of 
data they gather and analyse. For example, methods, such as the Talk-based Anal- 
ysis or Narrative Analysis focus on the analysis of talk and text, while others, such 
as Dramaturgical Analysis and Frame Analysis enable researchers to also take into 
account non-verbal forms of expression and performances. This part also includes 
examples of methods that sit at the intersection of quantitative and qualitative modes 
of analysis, such as Q methodology or Mixed Methods. 

Chapter 18 discusses the use of Ethnography in the study of deliberative democracy. 
Nicole Curato and Nicole Doerr outline two purposes of ethnography—to describe 
and to critique the lived experience of deliberative practice. The chapter presents a 
variety of empirical cases covering ten years of ethnographic work. In doing so, the 
chapter invites reflection on the positionality of the observer in culturally complex 
and multilingual deliberation. It also makes a case for conducting ethnography in 
non-ideal settings marked by inequality, to sharpen our understanding of deliberative 
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theory in relation to fields of visual and cultural sociology, performance, and affect 
studies. 

Chapter 19 introduces Rhetorical Criticism as a method that enables a close 
textual analysis of deliberative discourse. Influenced by poststructuralism, critical 
theory, and feminist theory, rhetorical criticism stresses how discourse constructs 
reality and how various material and symbolic contexts shape communicative prac- 
tices. Drawing on examples from recent scholarship, John Rountree outlines how 
rhetorical criticism can be undertaken. The chapter addresses the key research 
questions this method can help respond to, its units of analysis, its approach to 
building a dataset, and its method for translating textual evidence into interpretive 
arguments. 

Chapter 20 explores Process Tracing as a method for structuring qualitative, 
explanatory case-study analysis of deliberative processes. Jonathan Pickering out- 
lines process tracing as a method to identify causal mechanisms that connect the 
causes of events or phenomena to their outcomes, drawing evidence from a wide 
array of sources associated with a single case or a small number of cases. Many empir- 
ical studies in political science have employed process tracing in a loose manner, 
and only more recently have political scientists made a concerted effort to develop a 
more systematic approach. This chapter shows how deliberative democracy scholars 
can employ process tracing to assess deliberation within and beyond mini-publics, 
highlighting areas for future research. 

Chapter 21 presents Q Methodology, which is used for comprehending subjective 
viewpoints or discourses, involving both quantitative and qualitative elements. Lucy 
Parry shows how Q methodology is used to identify the balance of discourses in 
various sites witihin the broader deliberaitve system. This is illustrated in a detailed 
guide drawing on a Q methodology study on the representation of animals in the 
foxhunting discourse in the United Kingdom. 

Chapter 22 introduces Dramaturgical Analysis as a way of analysing the perfor- 
mative aspects of public deliberation. Selen Ercan and Carolyn Hendriks outline 
the key dimensions of dramaturgical analysis, such as scripting, setting, stag- 
ing and performance. The chapter shows how these dimensions can be used to 
analyse the communicative interactions in various settings, ranging from struc- 
tured forums to the broader public sphere. Dramaturgical analysis directs the 
researcher's attention to often-overlooked or taken-for-granted aspects of pub- 
lic deliberation, such as the performative styles and body language of the actors 
involved in deliberation, where they stand, how they enact and stage their argu- 
ments, what symbols and artefacts they use to reinforce their viewpoints, and how 
they reach out and persuade diverse audiences. Drawing on dramaturgical analysis, 
researchers can study verbal and non-verbal interactions taking place in deliberative 
practices. 

Chapter 23 explains how Narrative Analysis can be used in deliberative democracy 
research. John Boswell argues that while narrative analysis has much to offer scholars 
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of deliberation, the growing interest in and adoption of this approach in social science 
research presents some complexities and confusions. The chapter clarifies a version of 
narrative analysis considered as particularly suitable for studying deliberative prac- 
tice (grounded in the traditions of interpretive policy analysis) and offers examples 
of how this analysis can be undertaken. 

Chapter 24 introduces Frame Analysis as a method for studying deliberative 
democracy generally, and deliberative systems in particular. Ricardo Mendonga and 
Paula Simões argue that the method’s focus on the contextual dimension of meaning- 
making processes offers a path for the investigation of discursive clashes across time 
and space. The chapter raises the question of how frames produce particular percep- 
tions of reality and how deliberative democracy draws attention to who has the power 
to produce such frames. The chapter distinguishes between three traditions of frame 
analysis and illustrates what frame analysis can contribute to the study of deliberative 
democracy through a variety of examples. 

Chapter 25 describes Talk-based Analysis, which can be used for analysing speech, 
discourse, and rhetoric delivered in deliberative spaces. As Paromita Sanyal explains, 
this method provides a tool for the qualitative analysis of who says what and how, 
employing a semiotic approach that links speech and performance. The talk-based 
method is useful for examining the influence of social stratification and inequali- 
ties on public deliberations. Sanyal shows how she used this method in state-citizen 
discussions in constitutionally mandated village assemblies, gram sabha, in India. 
The analysis draws attention to how citizens talk to the state, as they voice demands 
or requests for public goods and personal benefits, complain about government 
negligence, and protest corruption and government inefficiencies. 

Chapter 26 explores Media Analysis to study deliberative democracy. Rousiley 
Maia and Tariq Choucair argue that the systemic turn in deliberative theory has 
invited a reconceptualization of argumentative exchange across different contexts 
and spaces. A systemic approach cannot afford to ignore interfaces between deliber- 
ation in institutional forums and more mundane discussions, and, consequentially, 
the neighbouring field of mass media and digital communication. This chapter offers 
a way of employing content analysis for researching mass media material and diver- 
sified online platforms. It presents different ways of using content analysis and 
blending it with other techniques to study deliberation at micro, macro and sytem 
levels. 

Chapter 27 outlines the use of Mixed Methods to analyse deliberative processes 
and argues that mixed methods are well-suited to grappling with deliberation’s com- 
plexity. Oliver Escobar reviews methodological foundations and outlines questions 
and puzzles where mixed methods can contribute to deliberative scholarship. The 
chapter also covers research design, data generation, analysis, and quality standards, 
while offering examples and concluding with a call to strengthen the mixed methods 
community of practice within the field of deliberative democracy. 

Chapter 28 highlights the importance of Case Study Research for the study of 
deliberative democracy. Stephen Elstub and Gianfranco Pomatto provide guidance 


18 Researching Deliberative Democracy 


on how to select cases and collect and analyse data in the field of deliberative democ- 
racy. In comparison with other methods, case studies have the advantage of delving 
into an individual case with the help of various methods and thus exploring it in 
depth. While it might be expected that case studies are particularly apt for exploring 
deliberative forums, the chapter shows how such explorations can teach us important 
lessons about how deliberation can be scaled up in deliberative systems. 


Part IV: Enacting Deliberation 


Finally, the fourth part focuses on the methods that bring deliberation to action 
through research. The methods draw on deliberative ideals for producing both 
knowledge and action. One of these methods, Deliberative Policy Analysis, for 
example, seeks to bring together a range of actors including citizens, politicians, and 
experts for formulating and implementing democratically legitimate policies. When 
researchers enact deliberation as part of their research or analysis, they work along- 
side participants and seek to improve deliberation in practice. The chapters included 
in this part of the book show how this may happen in both structured forums and 
the wider public sphere. They also reflect on some of the limitations and challenges 
of enacting deliberation in the context of a research process. 

Chapter 29 outlines Deliberative Policy Analysis (DPA) as an effective alternative 
approach to mainstream technocratic policy analysis. Hendrik Wagenaar presents 
two distinguishing characteristics of DPA: its focus on inclusive deliberation as a 
strategy of policy inquiry, and its orientation towards practice. While DPA accom- 
modates a range of interpretive methods, this chapter focuses on the relatively 
neglected analysis of practice. The analysis of practice requires a combination of suf- 
ficiently close-up ethnographic observation to allow the researcher to capture the 
deliberative practices, and an inductive theoretical rendition of these observations. 
The chapter shows how DPA can help to reveal the mundane practices, the hidden 
configurations of the process of policy formulation and implementation. 

Chapter 30 makes a case for using Action Research in the study of public deliber- 
ation. Kiran Cunningham and Lilian Muyomba-Tamale draw our attention to some 
of the key principles that action research shares with deliberative democracy. These 
entail inclusion, equity, and the goal of collective knowledge production. Action 
research, with its roots in feminist studies and critical theory, not only helps to inves- 
tigate the processes and impacts of deliberation, but also offers a method of enacting 
deliberation. Drawing on the example of Civic Engagement Action Plans (CEAPs) 
in Uganda, the authors show how deliberation and action research can go hand in 
hand and enable the inclusion of citizens and civil society actors into governmental 
decision-making processes. 

Chapter 31 introduces Community of Inquiry (Col) as a group interview method. 
Kei Nishiyama elaborates on the philosophical roots of the Col, which was orig- 
inally pioneered by pragmatist philosophers, such as John Dewey, as a group 
dialogue for reflective knowledge-construction. As a research method, Col enacts 
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deliberative ideals in practice in two important ways. First, it encourages reflec- 
tion and reason-giving in the context of the interview process. Second, it empha- 
sizes collaborative questioning and active listening, which help to minimize the 
power imbalances between an interviewer and interviewees and between inter- 
viewees. Nishiyama shows how he uses Col in conducting group interviews with 
children. 

Chapter 32 presents an innovative method, namely The Deliberative Camp, to gen- 
erate knowledge about social movements interactions, while also promoting such 
interactions in practice. Donatella della Porta and Andrea Felicetti elaborate on 
how researchers can co-organize Deliberative Camps along with activists, and how 
these camps can be used to shed light on dynamic practices performed by social 
movements and nurture their critical reflection about these practices. As a method 
of inquiry, the Deliberative Camp formalizes deliberations occurring within and 
between social movements and provides a powerful tool to deepen the comprehen- 
sion of the relationships between social movements and deliberative democracy. The 
chapter discusses the main ideas behind this method, presents the key steps required 
for its implementation, and outlines its potential contribution as well as limitations 
for research on social movements and deliberative democracy. 

Finally, in Chapter 33, Jane Mansbridge provides concluding remarks on con- 
temporary research in deliberative democracy. In this chapter, which is entited 
Mutual Need, she argues that the field requires and benefits from a close relation- 
ship between theory, practice, and empirical analysis. In her view, this is essential 
for the production of normatively legitimate decisions in a complex world. Mans- 
bridge concludes the book with a note of hope on human ingenuity and the capacity 
to strengthen democracy in the face of the grave problems confronting our societies 
today. 

We hope that this book captures the diversity of available methods, and we look 
forward to learning about other methods that we failed to include in this collection. 
Our aim in publishing this volume open access is to ‘practice what we preach. We 
want anyone wishing to learn about or take part in methodological debates in deliber- 
ative democracy to be able to do so without any prohibitions to access to information. 
We hope that readers find our book both useful and empowering. Rather than a 
canonical volume that closes a debate, we see this book as the start of a conversa- 
tion that will increase the diversity and sophistication of empirical and theoretical 
engagements in future deliberative democracy research. 
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PART | 
THEORIZING DELIBERATION 


2 
Methods of Theorizing 


Simone Chambers 


Deliberative democracy is a large and growing research paradigm. One of this 
paradigms defining features is the way it has attracted both empirically oriented 
scholars as well as theorists and philosophers. Although there is always disagreement 
about the best or most useful methodological paradigm, empirical social sciences 
have a relatively well-defined set of methods, tools, and approaches to work with 
when designing research projects. Indeed, methodology is usually a subfield within 
empirical social sciences. The question of methods and methodology is not so well 
defined in political theory where there is no clear toolbox. What is theory? What does 
it mean to do theory? What are the different ways to theorize? This chapter canvases 
some common answers to these questions and offers a typology of theory as it relates 
to deliberative democracy. 


What Is Theory? 


Empirical social sciences (indeed all sciences) employ and generate theory. This is 
often (but not always) understood as involving hypothesis construction and gener- 
alization. This type of theory can be very modest, for example a theory of recycling 
would develop and then test hypotheses about the sorts of factors that are likely to 
lead to recycling. But it can also produce theory on a grand scale. Modernization 
theory, for example, uses the same independent/dependent variable structure but 
on a macro scale to suggest a causal connection between economic development 
and democracy (Przeworski and Limongi 1997). While causal theories are the most 
common type of theory in empirical social science, they are not the only type. Max 
Weber had a theory about the way Protestant (Calvinist) theology, human psychol- 
ogy, and capitalist accumulation interacted, which postulated an ‘elective affinity’ 
between Protestantism and capitalism (Weber 2002). This theory contributed more 
to understanding the forces involved in modernity and processes of rationalization 
than to explaining the causes of capitalism. Weber did not test his theory but instead 
relied on a complex narrative about, and interpretation of, empirical phenomena as 
well as moral, psychological, and theological claims. 
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Deliberative democracy also generates various types of empirical theory, 
some falling squarely into the causal/testable type and others into the interpre- 
tive/descriptive type. For example, questions about the empirical conditions of 
persuasion can generate hypotheses and generalizations that can be tested in all sorts 
of ways (Bachtiger 2018). Or a systemic approach to deliberative democracy gener- 
ates the claim that spheres of communication can be or are connected and this in turn 
can generate a theory of communication flows that is primarily descriptive but not 
causal (Hendriks 2006). But these types of theories (causal and descriptive) are not 
what we usually mean by theory when we talk about deliberative democratic theory. 
One can generate empirical hypotheses out of the theory, but the theory itself is not 
primarily about how the world works or what is causing what; rather it is about how 
the world ought to work. 

Theory in political theory usually means normative theory. Normative theory is 
essentially prescriptive rather than descriptive (although a great deal of description 
can be included) and focuses on why we should value certain things (like delibera- 
tion) and what we ought to do to promote what we value. This normativity is to be 
understood very broadly and can be, and often is, deeply rooted in empirical real- 
ity. When we think about democratic theory more generally, in the first instance we 
think that theory outlines why democracy is preferable to other types of regimes, 
and second, what constitute the defining features of democracy that need to be pro- 
moted. Deliberative democratic theory starts here too and only at a second-order 
level does normative theory generate empirical types of theory. For example, if the 
theory of deliberative democracy claims that we can expect (predict) certain benefits 
from deliberation, then we should design research projects that ‘test’ such claims. 

So, by ‘theory’ I mean normative theory, and by ‘normative’ I mean theory gen- 
erating values, prescriptions, and criticisms rather than hypotheses, generalizations, 
explanations, and predictions.’ Before outlining what I take to be the main ways of 
engaging in normative theorizing, I want to say a few general words about methods. 


What Is a Method? 


Method and methodology are categories that can include many different things, from 
techniques, styles, and approaches to epistemological assumptions and philosophi- 
cal commitments. Here I will outline five very broad ways to understand the role and 
function of normative theory in the study of deliberative democracy. These five ways 
represent a middle position between, on the one hand, what might be called theories 
of the source of normativity and, on the other, techniques employed in the service of 


1 Treject the view that associates normative theory exclusively or primarily with the analytic tradition of 
political philosophy (List and Valentini 2016). Analytic political philosophy is more ofa style of philosophy 
than a method per se and is closely connected to the tradition of Rawls and his critics. 
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normative theory. In the first category, one might find discussions of Rawls’ reflexive 
equilibrium, Habermas's rational reconstruction, or Kant’s transcendental critique. 
These are all ways or methods, if you like, that particular thinkers have used to build 
their moral and political philosophies, which in turn have been drawn on by theories 
of deliberative democracy. As my focus is on deliberative democratic theory and not 
moral and political theory more generally, I will not spend much time discussing 
these foundational methods. At the other end of the spectrum are the specialized 
tools and techniques used in normative theory. Although I suggested above that theo- 
rists do not have the clear and well-organized toolbox of the empirical social scientist, 
there is a toolbox of sorts that contains such things as textual interpretation, the use 
of archives, ethnography, rational choice models, and counterfactual thought exper- 
iments (such as ‘trolley cars’) just to name a few. But tools and techniques do not 
add up to a full method or give one a sense of how to do normative theory. A ham- 
mer cannot tell you how to build a house and it certainly cannot tell you what kind of 
house you want to build or why you might want to build it in a particular way. So, the 
methods of normative theorizing with which I begin ask how we go about thinking 
about the sort of house we want to live in. 


Normative Theorizing 


In what follows, I divide normative theorizing into five categories. These are some- 
what artificial and analytic, and much of what we might call deliberative democratic 
theory engages in more than one of these and sometimes all five. Nonetheless these 
five categories can serve as a helpful heuristic and might be useful for people who 
are thinking through what they wish to accomplish with their contribution to theory 
and how to conceive of theoretical research projects. The first category I call theories 
of the ideal and these lay out what is deliberative democracy, normatively speaking, 
and why it is good or desirable. The second and third categories of theories look to 
be derivatives of the first (and they are) but they can sometimes make even more 
important contributions to the theory of deliberative democracy than theories of 
the ideal. The second category is critical reflective theory and this theory questions, 
probes, and interrogates the first-order theories of the ideal; the third category I call 
constructive reflective theory and here theorists interpret, elaborate, and extend the- 
ories of the ideal. The fourth and fifth categories engage the empirical world more 
directly, but like the reflective type either critically or constructively. Critical applied 
theory engages normativity by criticizing existing arrangements for failing to live up 
to normative ideals. Constructive applied theory engages normativity by suggesting, 
proposing, and designing innovative institutions and real-world practices to make 
democracy more deliberative, and so better. 
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Theories of the ideal are not to be confused with ideal theory. Ideal theory refers 
to a very specific type of theory (perhaps even a method) predominantly associated 
with the work of John Rawls in which, when thinking about principles and how they 
might work in the world, one imagines full compliance (Levy 2014). Furthermore, the 
normativity of theories of the ideal is not necessarily idealized in the sense invoked 
by realists. Realist criticism of some ‘normative’ theory is itself normative in the gen- 
eral sense I am using here. Realists argue that highly moralized or idealized political 
theory is deeply mistaken not because it is bad science (although it might be thought 
to rest on bad science) but because it leads us to misunderstand politics and so make 
disastrous decisions, choices, and evaluations. Realists from Machiavelli and Mor- 
genthau to Bernard Williams and Raymond Geuss all argue for realism on essentially 
normative grounds and put forward their own theories of the ideal, but here it is 
of course the realist ideal of a politics based on a true understanding of power and 
interests. 

Theories of the ideal contain two parts. They define what deliberative democracy 
is and they explain why we should value deliberative democracy. In this category 
we find what has come to be referred to as first-generation deliberative democracy, 
which includes the seminal work of Bernard Manin (1987), Jürgen Habermas (1989, 
1996), Joshua Cohen (1989), and Gutmann and Thompson (1996) to name only a 
few. It would be a mistake, however, to think of theories of the ideal as a genera- 
tional category and my typology of normative theory is not intended as a diachronic 
reconstruction of the holy history. First, there are theorists still contributing to this 
type of theory. Here I am thinking of the work, for example, of Rainer Forst (2009), 
John Dryzek (2017), and Mark Warren (2017). And second, there is a way to retroac- 
tively expand our repertoire of theories of the ideal. For example, we can go back and 
re-read theorists to find and extract these ideals. This is why the history of political 
philosophy is still a wonderful resource for theorists of deliberative democracy even 
though most people date the ‘birth’ of this research paradigm to somewhere in the 
1980s and 1990s. Two sources that come to mind are the re-appropriation of Aristotle 
on the wisdom of crowds and a re-reading of Dewey in light of deliberative democ- 
racy (Ober 2013; Knight and Johnson 2011). Neither Aristotle nor Dewey have clear 
theories of deliberative democracy, but they do contain ideals and values that can be 
recovered in the service of deliberation. 

It is hard to describe how to do this kind of theory. Its central reference point is 
democracy, and its core claims have to do with why deliberation offers a promising 
core ideal to think about, evaluate, and design democracy. Although the primary 
contribution of this type of theory is to articulate core ideals that animate the 
paradigm, the theory itself can be more or less abstract. If we compare the work of 
Joshua Cohen and John Dryzek, both of whom have powerful articulations of core 
features of the deliberative ideal, we see that their divergent ‘methods’ and styles can 
be tied to their respective research traditions. Cohen works in the analytic tradition 
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of Rawlsian political philosophy in which tightly constructed abstract argumentation 
defends reason-giving as the most coherent way to understand the value of free- 
dom and equality of each and every person. In contrast to Cohen, Dryzek comes 
out of a policy studies background. Here the style is much more problem-driven 
with closer links to empirical research. It nevertheless contains fundamental and 
original articulations of what is deliberative democracy (normatively speaking) and 
why deliberative democracy offers a more persuasive view of democratic legitimacy 
than aggregative views. 


Critical Reflective Theory 


Deliberative democracy as a research paradigm has grown and developed in quite 
unprecedented ways over the last thirty years. Part of this success has been the power 
of criticism (internal and external) to initiate important learning processes in the 
theory without exploding or destroying the paradigm. Critical reflective theory inter- 
rogates and questions core concepts first articulated in theories of the ideal. As noted 
at the outset, the lines between theories of the ideal and the two reflective types I 
discuss in this and the next section can be quite blurred: first, because some of the 
best critical reflective and constructive reflective theories contribute to theories of the 
ideal; and second, because theory in general is almost always in some sort of reflective 
relationship with predecessors and contemporaries. Despite some blurry lines, how- 
ever, I think having an intellectual past and acknowledging influences is not quite the 
same as engaging in critical reflective or constructive reflective theorizing. 

The paradigmatic case for me of critical reflective theory is found in the work of 
Iris Marion Young. Young questioned and criticized what she deemed to be a ratio- 
nalist bias in the conceptual core of deliberative theory and asked us all to think 
about pluralism, diversity, and difference in relation to deliberation (Young 2000). 
She brought the role of internal and external exclusions to the attention of deliber- 
ative theory. This was an enormously important contribution, indeed corrective, to 
deliberative theory undertaken through critique. The method was to imagine real- 
world deliberations governed by the narrowly conceived notion of reason-giving and 
to see the normatively undesirable consequences of the thought experiment. Thus, 
although Young challenged theories of the ideal from the point of view of real-world 
consequences, she did not use hard empirical data or historical events as evidence. 
She was not arguing that the theory was empirically false, but rather that it was nor- 
matively problematic: when thought through, it does not jibe with other normative 
commitments we hold strongly. 

I want to contrast Young's criticism of deliberative democracy with another much- 
cited critic. Lynn Sanders’ article, with the provocative title ‘Against Deliberation, 
points to a number of places and venues in the real world where deliberation is not 
equal and there are significant barriers to entry (Sanders 1997). Sanders’ essay, along 
with a mountain of empirical research cataloguing the way real-world conversations 
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fall short of the deliberative ideal, alert us to the pitfalls, challenges, and outright 
failures of deliberation to achieve the ideal of inclusive equality, but they do not con- 
tribute to normative theorizing in quite the same way as Young’s work does. Sanders 
shows that we are often not very good at actualizing the deliberative ideal and that 
there may be more obstacles to that actualization than we had anticipated, but she 
does not show that the ideal is not valuable; Young shows us that the way we had 
been conceptualizing the ideal might have normatively undesirable consequences 
and so we should rethink or be more careful and explicit about what we mean by 
reason-giving and be more sensitive to questions of inclusion. 

The critical reflective approach is a very common research design within norma- 
tive theory. Very generally there are three axes along which criticism can develop that 
focus on normative, conceptual, or empirical adequacy of the theories of the ideal. 
Iris Marion Young is an example of the normative type. There is a lot of work in phi- 
losophy and political theory that interrogates the conceptual adequacy of Rawlsian 
and Habermasian philosophy. Only some of that research is relevant to deliberative 
democracy and identifying what is and is not relevant is a matter of taste. Pub- 
lic reason, for example, is a concept that has generated a whole subfield of critical 
reflective theorizing, much of it focused on conceptual adequacy. How much of that 
debate can be considered part of deliberative democratic theory depends on how one 
understands deliberative democracy. 

As I suggested above, the question of the empirical adequacy of normative theo- 
ries is a tricky one. On the one hand, normative theories do not claim to describe 
the world, but rather invite us to think about how the world ought to be. So, sim- 
ply pointing to the ways in which the real world fails to live up to the ideals of 
deliberation does not necessarily add up to a serious question of adequacy. This con- 
fusion between descriptive and prescriptive claims is a very common mistake made 
by empirical social scientists who are intent on criticizing the empirical adequacy of 
normative theory (Achen and Bartels 2016). On the other hand, a normative theory 
that is based on wildly implausible empirical assumptions indeed has an adequacy 
problem. I do not think that theories of the deliberative democracy ideal are based 
on wildly implausible empirical assumptions, but I do think that early misreadings 
of theories of the ideal presented versions (sometimes caricatures) of the ideals that 
did seem deeply implausible (Mouffe 2000; Geuss 2008). I discuss this in more detail 
below. 


Constructive Reflective Theory 


The idea of constructive reflective theory is to build on, elaborate, and extend 
theories of the ideal. This can be done at the same time as engaging in critical 
reflective work, of course. James Bohman is a good example of a theorist who, in 
critically interrogating the concept of consensus in Habermas, has extended and 
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elaborated the Habermasian ideal by developing the concept of ‘plural agreement’ 
as a more pluralism-friendly conception of the ends of deliberation (Bohman 1996, 
34). Another good example of this type of theory can be seen in the work of Cheryl 
Mizak (2008) who takes up and defends the work of the pragmatist Charles Sanders 
Peirce and develops his idea of ‘inquiry’ into an epistemic defence of deliberative 
democracy. I see much of my own work as an elaboration, extension, and application 
of Habermas's model of two-track democracy to contemporary issues and contexts 
(Chambers 2017). Finally, the most successful and perhaps paradigmatic version of 
this type of theory is the systems approach to deliberative democracy (Mansbridge 
et al. 2012). 

Somewhat dissatisfied with the growing domination of small-scale one-off delib- 
erative venues as the focal point of much deliberative democracy research, many 
theorists have sought to scale up the analysis of deliberate democracy (or reinvigo- 
rate the scaled-up analysis found in thinkers like Habermas) into a systems approach. 
The challenge has been to re-imagine the core normative values of deliberation 
(inclusion, equality, reason-giving) at a macro scale while avoiding simplistic and 
unrealistic views of democracy as one big face-to-face deliberation. André Bachtiger 
and John Parkinson articulate the core intuitions of the systems approach very 
well: 


Starting from the view that no one event or process can possibly maximize all the 
deliberative democratic criteria—it cannot be maximally inclusive, decisive and 
deliberative all at once—the systems approach assumes the necessity of a division 
of labour between institutions and actors with different functional strengths and 
weaknesses. 

(Bachtiger and Parkinson 2018, 85) 


The systems approach has produced a new wave of normative theory that has 
been informing the next two types of theory. But the systems approach has been 
the focus of several important critical reflective studies as well. David Owen and 
Graham Smith’s important challenge to systems theory is a good example (Owen 
and Smith 2015). Like Iris Marion Young, they ask us to imagine some of the norma- 
tive consequences of the systems approach and in particular the division of labour 
thesis. Here they suggest that, if we are not careful, normative theory will go down a 
functionalist rabbit hole whereby deliberative outcomes might seem to be the prod- 
uct of systemic forces and yet no individuals actually deliberate, and the picture at 
the individual level might look quite grim. Again, like Young, this challenge has not 
demolished the normative claims of the systems approach (nor was it intended to). 
It has instead spurred a deeper engagement with the normative core of deliberation, 
particularly the question of the ethical requirements of citizens in a well-functioning 
deliberative system. 
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Critical Applied Theory 


The critical applied model of normative theorizing takes the normative theory gener- 
ated by the three types of theory outlined above and uses these theories as yardsticks 
in assessing and criticizing arrangements in real existing democracies. While the foci 
of reflective theory are other theories, the focus of applied theory is the real world. 

With the expansion and growth of deliberative democracy as a research paradigm, 
and the successful collaboration between theory and empirical social sciences, 
applied types of normative theorizing are perhaps the fastest growing field today. The 
major challenge is in operationalizing normative ideals into either a critical yardstick 
or a useful and workable institution. This turns out to be very tricky. In this section 
I focus on the yardstick type of normative theory. 

Habermas is well aware of the pitfalls involved in translating normative ideals into 
workable pragmatic categories. Many theorists mistakenly thought that Habermas’s 
moral theory, discourse ethics, was a normative political theory to be directly imple- 
mented in some way in the real world. As he clarifies in Between Facts and Norms, 
however, ‘An unmediated application of discourse ethics (or of an unclarified concept 
of discourse) to the democratic process leads to muddled analysis; these then offer 
skeptics pretexts for discrediting the project of a discourse theory of law and politics 
at its inception (Habermas 1996, 158). Critics and enthusiasts alike often appealed 
to Habermas in projecting a highly utopian political theory in which open-minded 
and rational citizens would reach cooperative consensus on tough political questions 
under egalitarian conditions (Geuss 2008). Agonistic theorists like Chantal Mouffe 
accused deliberativists of wanting to reduce all politics to a rational search for con- 
sensus (Mouffe 2000). The mistake here was to think that the ideal speech situation 
offered a simple yardstick against which to compare all political communication (not 
to mention all social relations). But the ideal speech situation is a device to represent 
and model some essential aspects of discourse and justification. Habermas has long 
abandoned reference to it precisely because so many have read it as containing a goal 
(even if an unreachable one) to be striven for. It offers insight into when and where 
processes of communication and justification have gone off the rails. In this sense, 
then, it is a tool of critique and not an end to be achieved. Some normative ideals are 
analogous to the idea of a vacuum in science. Thinking about the movement of bod- 
ies in a vacuum can tell us a lot about the forces that affect the movements of bodies 
in the real world and can even suggest ways to improve the movements of bodies, but 
this useful idealization is not intended as a place to live. 

Deliberative democracy offers many tools of critique and Habermas's discourse 
theory is only one source. The objects of critique can vary from a critical analysis of 
a single referendum (Chambers 2018) or protest event (Mendoça and Ercan 2015) 
to democratic regimes more generally (Curato et al. 2019). Although these studies 
often contain a great deal of empirical description, analysis, and interpretation, they 
are also contributions to normative theorizing because in operationalizing norms we 
can get a better understanding and articulation of their meaning and significance. 
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The yardstick model is critical in the sense that it interrogates, evaluates, and analyses 
real-world arrangements from the normative point of view, but it does not always 
have to be critical in the negative sense. It is possible, especially in a comparative 
framework, to point to positive developments and good practices. But in an imperfect 
world beset by power, interest, and money, it stands to reason that the real world will 
always fall short even of our mediated realistic ideals. Deliberative democracy is not a 
utopian theory with a blueprint of the just and fully democratic society of the future. 
For deliberative democrats, progress (if it occurs at all) is piecemeal, incremental, and 
often achieved by pointing out the egregious failures of institutions and people to live 
up to the ideals of deliberative democracy. 


Applied Constructive Theory 


This last category of theory is in some sense the least theoretical (but not in a bad 
way), and it is the area that has seen (and I predict will continue to see) the great- 
est progress. It also involves operationalizing theoretical norms but the task here 
is somewhat different than what we saw in the critical applied model. As I noted 
in closing the last section, deliberative democracy is not a theory with a utopian 
blueprint. It is not an alternative to representative democracy although it does sug- 
gest many institutional innovations to enhance and perhaps, over the long run, 
radically restructure representative democracy. We are seeing, for example, exciting 
suggestions for bicameral assemblies that adopt sortition as the method of selection 
and turn representative assemblies into citizen assemblies (Gastil and Wright 2019). 
I close this chapter with three examples of applied constructive theory that not only 
operationalize deliberative norms in tangible and useful ways, but also reflexively 
contribute to the normative theory on which they draw. 

James Fishkin has been refining the Deliberative Poll for almost thirty years 
(Fishkin 2018). The focus has been on operationalizing norms of deliberation that 
enhance the epistemic quality of citizens’ opinions, as well as procedures of random 
sampling, in order to make good on the normative claim that deliberative polls can 
be understood as representative of the larger public. Such careful attention to design 
features has given us a huge amount of empirical evidence about the conditions 
under which citizens make evidence-driven, cognitively sophisticated judgements 
and offers a serious challenge to those claiming that citizens are fundamentally 
incompetent reasoners. 

John Gastil has been a champion, designer, and student of the Citizen Initiative 
Review (CIR) (Gastil et al. 2016). Whereas Fishkin, until recently, has concentrated 
on operationalizing the ideal conditions of deliberations, Gastil has been thinking 
about the connections to be made between citizen deliberation and the broader pub- 
lic debate, thus operationalizing norms of the systems approach. The CIR was created 
to help citizens outside citizens’ assemblies make more informed choices on state- 
wide ballot measures. Between nineteen and twenty-four citizen panellists, drawn 
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from a stratified random sample, convene to deliberate for four to five days about 
ballot initiatives. They hear from both sides of the issue, talk with neutral witnesses, 
deliberate intensively as a full panel and in small break-out groups, and then write 
a one-page analysis for distribution to the wider electorate. Data shows that when 
citizens are aware of what the CIR is and does, trust in the information increases and 
may outweigh that of partisan sources (Gastil et al. 2017). 

The work of Farrell et al. explores the use of deliberative institutions (both full 
citizens’ assemblies and partial citizens’ assemblies) in processes of constitutional 
reform in Ireland (Farrell et al. 2018). The work, while very empirical, begins from a 
conviction that deliberative democratic theory contains promising suggestions as to 
how to address some pressing and difficult questions facing democracies. Their work 
offers empirical evidence of some success in this venture and there is also evidence 
that other jurisdictions are looking to emulate the Irish experiment. 

The three research initiatives outlined above, all focus on designing and inserting 
randomly selected citizens’ mini-publics into the broader repertoire of representative 
democracy. These are very exciting applications of normative theory and although 
many of the scholars here are not ‘normative theorists, their work draws on and 
applies normative theory, reflexively contributes to normative theory, and most 
importantly, explicitly takes up the normative stance of endorsing deliberative insti- 
tutions as something we ought to be promoting. There are many more ways to engage 
the empirical world via deliberative democracy than the two types of normative 
applied theory I have just outlined. The rest of this book canvases many of those ways 
and describes a number of fruitful methods to employ. All of them have normative 
theory somewhere in the background even though not all of them directly engage in 
or with normative theory. And all of them can reflexively contribute to normative 
theory. 
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3 
Formal Models 


James Johnson 


How can formal models help us understand deliberative institutions? Many politi- 
cal theorists might well be tempted to answer that question simply and firmly: “They 
cannot. Indeed, it hardly is uncharitable to state that ‘the normative literature on 
deliberation, with very few exceptions, takes essentially no account of the presence of 
the game-theoretic work on deliberation (Landa and Meirowitz 2009). The problem 
with this state of affairs is that it overlooks decades of productive intellectual engage- 
ment. This is not the place for a lengthy review of that history. List (2018) offers an 
excellent, detailed, and comprehensive overview. 

It is nevertheless important to address the aversion some ‘deliberative democrats’ 
display towards formal models. That aversion is misguided, an artefact of a tangle 
of conceptual commitments that informed early defences of deliberative democracy. 
Those early defences traded on what, following Dewey (1920, xxxi), we can call a 
‘whole brood and nest of dualisms. Theorists spoke of deliberation versus aggrega- 
tion, of communicative versus strategic action, of reasonable versus rational grounds 
for judgement and action, of arguing versus bargaining, and of ideal versus non-ideal 
theory. These dualisms were unsustainable and consequently led inquiry astray. They 
have been only slowly and partially relinquished.’ As a result, advocates (and critics) 
of deliberative democracy failed to notice that purveyors of formal models had all 
along recognized the importance of discussion, debate, and argument in politics. 

The recognition to which I refer emerges more or less from the start. Having proved 
his famous impossibility result, Arrow (1963) immediately explains that the existence 
of single-peaked preferences amongst a constituency affords a possible escape from 
what some democratic theorists take to be a dire fate. Single-peakedness is the most 
familiar of several ‘value restricted’ preference orderings (Sen 1966). Such restric- 
tions do not require substantive agreement across individuals as to how the issue 
should be decided. Rather, they impose a particular structure on individual orderings 


* Many of my early papers were aimed at deflating these dichotomies (Johnson 1991, 1993, 1998; Knight 
and Johnson 1994). While some advocates of deliberation apparently remain committed to this tangle of 
unsound dichotomies (e.g. Steiner 2008), numerous others eventually came to accept the arguments I and 
others had been making all along (e.g. Mansbridge et al. 2010; Warren and Mansbridge et al. 2015). This is 
the moment to acknowledge that my thinking on the matters I discuss here has been profoundly influenced 
by my collaboration with Jack Knight (Knight and Johnson 2007, 2011). He is not, however, responsible 
for any of the formulations I offer here. 
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and in that sense suggest that relevant agents share a common understanding of the 
decision confronting them. Crucially, they do not show Arrow’s Theorem to be mis- 
taken. Admitting such ‘value restricted’ preference orderings violates one of his initial 
axioms, namely universal domain. 

I return to this idea shortly. Here it is important only to note that Arrow him- 
self insists—invoking in the process ‘the works of Rousseau, Kant, T. H. Green, and 
many others’—that ‘the case for democracy rests on the argument that free discus- 
sion and expression of opinion’ are amongst the most effective and legitimate means 
of promulgating value restricted preferences. This, of course, converges with the 
now common observation that, historically speaking, in representative democracies 
voting typically is preceded by argument or debate (Manin 1997). The outstanding 
questions are whether and how such processes of communication relate to some more 
idealized concept of deliberation. 

Political scientists commonly use two families of formal model—game theory and 
social choice theory. No bright line demarcates the two methods. Austen-Smith and 
Banks (1998, 1999, 2005) and McCarty and Meirowitz (2007) document the canon- 
ical works. Unlike the methods discussed by other chapters in this volume, formal 
models are not directly empirical. Instead, as I note below, we use them for concep- 
tual purposes. That said, formal models are crucially important to those engaged in 
empirical political inquiry generally and to empirical studies of deliberation in par- 
ticular. Diana Mutz makes this clear in her brief for ‘getting normative theory and 
empirical research to speak to one another in the realm of deliberative theory’ She 
suggests that theories of deliberative democracy are deficient in several ways that are 
obstacles to such an intellectual undertaking. Most importantly, she suggests that 
deliberative theories lack clearly defined concepts and are inattentive to the ways that 
their various component concepts relate to one another (Mutz 2008, 224). Absent 
such conceptual clarity, Mutz contends, it is difficult to know how to even proceed 
in formulating empirical assessments of theories of deliberative democracy. In what 
follows I argue that formal models are useful to students of deliberative democracy 
insofar as they help provide the sort of conceptual clarity that empirical assessments 
presuppose. 

Mutz rightly complains that, in the broadest sense, ‘it would be a stretch to claim 
that deliberation is itself a well-defined concept’ (2008, 225). Do we think of demo- 
cratic deliberation in moralized terms? Do we insist, in other words, that what 
counts as ‘deliberation’ is constituted by the qualities of individuals and their com- 
munication (e.g. their other-regardingness, receptiveness to competing viewpoints, 
willingness to change their minds, etc.)? Or do we treat ‘deliberation as debate, 
testimony, and other sorts of political communication conducted under certain nor- 
mative constraints (e.g. freedom and equality)? I have argued for the latter view 
(Knight and Johnson 2007, 2011). Recent work by formal theorists goes a consid- 
erable distance towards clarifying this view (Landa and Meirowitz 2009; Chung and 
Duggan 2020). 
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Rather than pursue that broad line of inquiry, I here want to take Mutz’s advice 
that we approach the empirical assessment of theories of democratic deliberation by 
focusing on ‘middle range’ concerns (2008, 522). I will focus on how formal models 
can help us to understand one particular and notably neglected concept: the idea 
of ‘agreement.’ Before we address that crucial substantive question, it will help to 
briefly consider how we should think about models and how we use them. It may be 
that political theorists writing on democratic deliberation imagine that models are 
an inapt tool for normative inquiry. That, it turns out, is seriously misguided. 


Models—What Are They Good For? 


Prominent political theorists rely on models at crucial junctures in their arguments. 
This is true, for instance, of both Rawls and Foucault (as well as those labouring in 
their shadows). For Rawls think of the Original Position. For Foucault think of the 
Panopticon. Moreover, they rely on those models in the same way, namely for concep- 
tual purposes (Johnson 2014). Rawls and Foucault use models to show how, starting 
from very abstract concepts of justice and power respectively, they arrive at their 
more concrete conceptions of justice as fairness and disciplinary power.’ That, as it 
turns out, is how democratic theorists concerned with deliberative institutions use 
formal models as well. 

Here I depart from the prevailing view in the discipline regarding how and why 
we use formal models. According to what I call ‘the standard rationale’, we use such 
models for more or less directly empirical purposes. We use them to derive predic- 
tions that, treated as empirical claims, we can test as hypotheses in familiar ways. 
Unfortunately, this conventional understanding turns out to be thoroughly mislead- 
ing. Like Rawls and Foucault, we rely on formal models for conceptual purposes—to 
articulate what we mean by such abstract concepts of ‘rationality; ‘institution, ‘deter- 
rence, and so on in concrete circumstances (Johnson 2019, 2021, 2022). In what 
follows I focus on how formal models help us articulate a conception of ‘agreement’ 
that is both normatively attractive and analytically useful for inquiry into democratic 
deliberation. 

Before turning our attention to deliberative institutions, it is clearly important to 
answer an obvious question: What is an institution? An institution is a set of rules 
(e.g. roles, procedures, offices) that provides enduring means of coordinating our 


? In short order, the Annual Review of Political Science published four essays surveying the growth indus- 
try in research assessing the empirical performance of theories of deliberation. See Delli Carpini et al. 
(2004), Ryfe (2005), Mutz (2008), and Thompson (2008). See likewise Bachtiger and Hangartner (2010). 
None of these surveys so much as takes up the issue of how we might best conceptualize the agreement, if 
any, that deliberation might generate. 

ĉ Throughout I rely on the distinction between abstract or general concepts and more particular and 
concrete conceptions that will be familiar from reading Rawls and Dworkin. 
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social and political-economic interactions over time. Typically, institutions have a 
systemic quality, such that in any given setting an institutional arrangement will hang 
together in a more or less coherent, if arbitrary and contested, way. 

Institutions often require that individuals, groups and organizations act in ways 
contrary to what they might consider to be their own immediate or even longer-term 
interests, commitments, and attachments. In short, they trade not solely in voluntary 
compliance but in compulsion. They must therefore specify more or less fully what 
can be done, by and to whom, for what purposes, and when. They must also indicate 
what happens when the rules are breached and who decides when they are. Institu- 
tions are pervasive and crucially important. That much is a platitude. The difficulty 
for purposes of inquiry is that we cannot directly observe an institution per se. As 
Douglass North puts it: “We cannot see, feel, touch, or even measure institutions’ 
(1990, 107). We are, in other words, on properly theoretical terrain. An institution is 
a conceptual entity. 

In an abstract sense, institutions emerge from ongoing social and political interac- 
tion and subsequently structure those interactions. But this generic concept encapsu- 
lates divergent ideas. This is where formal models are especially useful. Specifically, 
they allow us to articulate at least two distinctive conceptions—institutional equi- 
libria and equilibrium institutions—that address divergent problems and, in the 
process, illuminate dimensions of the abstract concept ‘institution. 

A thumbnail intellectual history may help see this. McKelvey (1976, 1979) and 
Schofield (1978) demonstrated that in institutionally sparse spatial contexts, collec- 
tive choice is generically unstable. These are commonly called ‘Chaos Theorems. In 
reply, Shepsle (1979) advanced his conception of ‘Structure-Induced Equilibrium to 
show how, in legislative settings (but not only there), institutional rules constrain 
indeterminacy and coordinate collective choice on outcomes that are otherwise not 
equilibria. In short, he established one variety of institutional equilibrium. His argu- 
ment is helpful here because it shows the crucial impact of separating or collapsing 
dimensions of decision and action. 

The problem, of course, is that such institutional mechanisms are susceptible to 
being unsettled by relevant actors should they find themselves consistently disadvan- 
taged by them (Riker 1980). This claim is troubling insofar as our abstract concept 
of institution presumes that they persist over time. Calvert (1995a, 1995b) elabo- 
rated game-theoretic models to show institutions to be equilibria in some underlying 
repeated game and therefore less susceptible to being upset than might have been 
thought. In other words, he provides us with a conception of equilibrium institutions 
to account for how institutions emerge, change, or persist. None of these models is 
directly empirical. Most especially, none generates predictions in the way the stan- 
dard rationale presupposes. Instead, as a group, they sustain the claim that we use 
formal models for conceptual purposes (Johnson 2021). They help us to articulate 
and move from the abstract concept of institution to more particular conceptions of 
institution. 
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Structured Disagreement 


In our argument for the priority of democracy, Jack Knight and I rely on the results 
produced by formal models in several ways. We do not actually create any such mod- 
els ourselves (Knight and Johnson 2007, 2011). Instead, we are directly indebted to 
Dasgupta and Maskin (2004, 2008), for instance, who establish the ‘robustness’ of 
majority rule relative to other aggregation schemes. Here, though, it is more useful 
to take up an example raised earlier—the idea of value restricted preference order- 
ings and its usefulness for clarifying the concept of ‘agreement’ which is central to 
discussions of democratic deliberation. We rely on formal modes at this juncture as 
well. 

Any theory of democracy starts from some vision of what we call the ‘circum- 
stances of politics. We take those circumstances to be constituted by inescapable 
interdependence and irreducible pluralism (Knight and Johnson 2011, 1-3). On 
the one hand, members of a polity are stuck with one another. On the other, they 
typically articulate an exceedingly diverse range of material interests, moral commit- 
ments, and cultural identities. While the first feature necessitates that they identify 
ways to coordinate their ongoing interactions, the second ensures that they will 
disagree about the best ways of doing so. This predicament haunts not just deci- 
sions about policy but, more importantly, politics surrounding the emergence and 
change of institutions. How are we to understand whatever agreement prevails in 
these circumstances? 

It is implausible to imagine that consensus, meaning unanimity or anything even 
close to it, will emerge from the circumstances of politics. Moreover, for democratic 
theorists the task arguably is not simply to acknowledge diversity but to identify insti- 
tutional arrangements that actively sustain it. Here, formal models are extremely 
helpful. As mentioned above, a familiar way to address the impossibility Arrow 
reveals to hold of all aggregation mechanisms is to incorporate ‘value restricted’ 
preference orderings.* In the current context, the most commonly invoked sort of 
restriction is known as single-peakedness. 

Perhaps the best way to depict this condition is to present it in spatial terms. It 
helps us understand the familiar depiction of politics as occurring along a left-right 
dimension. In this sense left and right simply afford a useful and enduring ‘politi- 
cal topology’. They ‘are not ontological concepts but are instead two spatial concepts 
without specific and constant content’ (Bobbio 1996, 58, 56). Formal models help 
us specify what particular conceptions of left and right mean in relatively concrete 
circumstances. 


* As noted earlier, Arrow (1963) suggests this ongoing line of inquiry. Subsequent relevant studies 
include Sen (1966), Niemi (1969), Riker (1982), Miller (1992), Knight and Johnson (1994), List (2002), 
Dryzek and List (2003), Penn et al. (2011), List et al. (2013), Ottonelli and Porello (2013), Duggan (2016), 
and Rad and Roy (2021). One caveat is in order. Although this is not the place to pursue the matter, it is 
important not to adopt a caricatured idea of ‘preference. On this, see Hausman (2011). 
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Consider a set of alternatives arrayed across a horizontal axis from, to use common 
parlance, left to right. Then depict the order of preference for each actor on a verti- 
cal axis. Roughly, imagine each individual to have an ideal point—a most preferred 
alternative—amongst some set of available alternatives (e.g. candidates, policies, 
etc.). Her preference ordering is single peaked if, starting at her most preferred alter- 
native, the lines connecting it to her second, third, and so on most preferred, until we 
reach her least preferred alternative, slope downwards (Riker 1982, 123-128). This 
can be generalized to allow for individuals to be indifferent between alternatives. 
Very recent work on deliberation focuses on the implications of this (Rad and Roy 
2021). In any case, a situation where the preferences of relevant individuals (voters) 
are so characterized commonly is interpreted to mean that ‘the voters have a common 
view of the political situation, although they may differ widely on their judgments’ 
(Riker 1982, 126). In short, in such situations, voters who may differ on substantive 
matters nonetheless share a common understanding of what is at issue in a political 
interaction. 

Questions arise immediately. What kind of agreement are we talking about here? 
How much of the ‘relevant’ population needs to ‘agree?’ Or, conversely, how much 
room remains for disagreement in deliberative politics? Consider each question in 
turn. 

As List (2002) succinctly puts it, there are ‘two concepts of agreement’ at work 
here. There is the familiar sort of substantive agreement or otherwise on policies or 
candidates. But there is also the sort of second-order or ‘meta agreement towards 
which Riker gestures. It establishes what is at stake in our first-order agreement or 
disagreements. It establishes the meaning, in this or that particular concrete context, 
of the generic left-right dimension over which our agreement or disagreement is 
arrayed. Even if we agree on this ‘meta level, we might well disagree—sometimes 
quite vehemently—at the first-order or substantive level. 

At least potentially, the sort of second-order agreement we are discussing allows 
an escape from the impossibility Arrow identifies. Potentially. To see what is at stake 
here, recall that it is well known that in circumstances where there are multiple 
dimensions, value restrictions like single-peakedness do not constrain the sort of 
pervasive collective indeterminacy Arrow illuminates. This is one pillar of the gen- 
eral indeterminacy of majority rule. But even if ‘discussion, debate, civic education, 
and political socialization (Riker 1982, 128) fail to reduce disagreement to a single 
dimension, political-economic institutions will often do just that. Recall the mod- 
els of institutional equilibrium I mentioned earlier. This, of course, says nothing 
about how extensive such second-order agreement must be if we hope to evade the 
instability of aggregation rules. 

Here we can turn to Niemi (1969), a remarkable paper that addresses precisely 
that matter. Niemi established that the demand that ‘all’ voters must have single- 
peaked preferences is overly restrictive. He recognizes at the outset that ‘complete 
unidimensionality is most unlikely empirically. But he demonstrates that we need 
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significantly less than total consensus. In large constituencies, collective intransitiv- 
ities ‘will infrequently occur if only 75% or 70% or even fewer of the individuals 
adopt a common standard (dimension) of judgment’ As a result, neither deliberation 
nor any of the other factors towards which he, like Riker, gestures need ‘accomplish 
the nearly impossible task of creating unanimous agreement on standards of judg- 
ment. Aggregate instability ‘can be very satisfactorily avoided if common frames of 
reference are widespread but far less than unanimous’ (Niemi 1969, 494). 

Several things follow here. First, there is a broadly methodological issue. Mod- 
els of the sort I have discussed—whether of the informal sort we find in Rawls 
and Foucault or of the formal sort exemplified by Arrow’s Theorem—are not tools 
for representing empirical features of politics. They are instruments of interpreta- 
tion. We use them for conceptual purposes. My focus has been on how we rely on 
formal models to interpret one concept central to any theory of democratic delib- 
eration: agreement. On this view, we use our formal ‘models [to] mediate between 
theory and the world’ (Cartwright 1999, 179-180; see also Johnson 2019, 2021). We 
use interpretive models to show, quite precisely, what, in this setting and for these 
purposes, the abstract concept of agreement ‘amounts to’ or ‘consists in—what it 
means—and to consider what rides on our doing so (Cartwright 1999, 41, 189). This 
need to ‘fit out’ abstract concepts in terms of more concrete conceptions in this way 
‘marks an entirely commonplace feature of language’ (Cartwright 1999, 40). But, 
more specifically, on this account scientific inquiry necessarily involves formulating 
testable theories that incorporate ‘abstract concepts ... whose relation to the world 
must be mediated by more concrete concepts’ I refer to those concrete instances as 
conceptions. These are ‘very specific in their form: the forms are given by the inter- 
pretive models of the theory. Following Cartwright, this understanding of models 
illuminates that they play an essential role in how our theories allow us to make sub- 
stantive claims about and, more importantly from her perspective, intervene in the 
world (Johnson 2021, 2022). 

Second, from a normative perspective formal models make clear that to the extent 
that it is informed by this conception of agreement, deliberative politics can accom- 
modate significant levels of pluralism. Not only can relevant individuals disagree 
vigorously about substantive, first-order matters of whether and how to address the 
matter at hand, but also nearly a third can depart from some shared understanding 
of what is at stake in their political interaction. That is good news. What we have is 
less a demand for agreement than a recognition that deliberation is amongst the fac- 
tors that structure our disagreements in ways that render them tractable in the sense 
that it is possible that they can issue in stable (transitive) aggregate outcomes. This 
is hardly a naive process. But it holds out the possibility that we can resolve or settle 
disagreements in justifiable ways (Knight and Johnson 2007, 2011). 

Finally, there is an analytical point. We are treating deliberation as a particular 
form of discussion or debate, one constrained by a set of normative commitments 
(equality, freedom, etc.). Not just any casting and counting of votes constitutes an 
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election. So, too, not just any exchange of ideas or proposals counts as deliberation. 
In this sense, relying on formal models points in the direction of a less moralistic 
view of deliberation. What differentiates deliberation from other forms of discussion 
and debate is not the virtues or otherwise of political actors or the type of speech acts 
in which they engage. Instead, what sustains that differentiation are the normative 
constraints—commitments to freedom, equality, and so forth—that are built into our 
democratic institutional arrangements. This fundamental shift underwrites what I 
take to be the most insightful recent contributions to thinking about deliberation and 
alternatives to it (Landa and Meirowitz 2009; Chung and Duggan 2020; Knight and 
Schwartzberg 2020). All of these papers focus our attention on political institutions 
and the normative characteristics they embody or fail to embody.” This, I believe, is 
wholly salutary. 


Conclusion: Lessons, Methodological and Substantive 


The argument I have sketched here suggests numerous lessons. I here mention just 
three. 

First Lesson: Embrace a basic pragmatist impulse. Beware of dichotomies and how 
they can enshroud inquiry (Putnam 2004). In the domain of political theory, it is 
especially important to resist the now conventional view that there is a dichotomy 
between facts and values and that this dichotomy sustains a clear and sharp distinc- 
tion between ‘positive’ and ‘normative’ inquiry. Given the pervasive ways facts and 
values are entangled, such a gerrymander is implausible. At least those seeking to 
police that boundary have yet to defend it persuasively (Knight and Johnson 2015). 
We should not, in other words, be sceptical of models, formal or otherwise, simply 
because some mistakenly imagine that they are the provenance of an enterprise called 
‘positive’ political theory. 

Second Lesson: Embrace yet another pragmatist impulse. Recognize that there is 
no sharp distinction to be drawn between models that are ‘merely’ verbal and those 
couched in mathematics. Instead, casting our inquiries in mathematical terms is con- 
tinuous with, rather than sharply distinct from, doing so in everyday language. What 
Quine calls the ‘mathematization of inquiry consists just in the ‘progressive sharpen- 
ing and regimenting of ordinary idioms’ (Quine 1986, 150). We should not, in other 
words, be sceptical of formal models simply because they are expressed in mathe- 
matical language. We use models like the Original Position and the Panopticon for 
basically the same purposes that we rely on game theory or social choice theory: 


° The relative indifference that theorists of deliberative democracy often display towards institutional 
mechanisms follows from accepting the common but ultimately misleading and unsustainable dualism 
between ‘ideal’ and ‘non-ideal’ theory. On this view, political theorists must first concentrate on articulat- 
ing principles and ideals and only then concern themselves with problems of how to approximate them in 
the world. This dichotomy is one methodological bequest of John Rawls. Not just his followers but many 
of his critics embrace it. It obstructs inquiry insofar as it relegates concern for institutions—in short, for 
matters of compliance and implementation—to a subsidiary role. Perhaps the most glaring problem with 
this advice is that Rawls himself never managed to follow it (Orr and Johnson 2018a). 
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conceptual exploration. That is the thrust of my discussion of how models illuminate 
‘structured disagreement. The idiom—verbal or mathematical—in which we cast the 
model is contingent. 

Third Lesson: We learn from formal models in multiple ways. We can, as many 
prominent political economists do, learn from formal models the crucial importance 
of precisely what they initially exclude. Think here of Thomas Schelling (1960) on the 
role of what he terms ‘incidental detail’ in underwriting the salience of focal points for 
individuals facing coordination problems. Or think of the principles Elinor Ostrom 
(1990) derives for discriminating more or less robust forms of ‘self-governance’ 
for groups confronting commons problems. Or, finally, think of the role Amartya 
Sen ascribes to ‘public reason’ in democratic politics beset by various problems of 
collective choice (Orr and Johnson 2018b). 

More obviously, perhaps, we can model processes of democratic politics them- 
selves as do provocative recent papers on debate and deliberation (e.g. Chung and 
Duggan 2020). But we can also, as do Knight and I, make inferences from formal 
models of, say, aggregation to features central to other domains of democratic pol- 
itics, such as debate and deliberation. That is precisely how we formulate our views 
about structured disagreement I have sketched here. And that, I suggest, advances 
our understanding, not just of deliberative institutions, but of democratic institutions 
generally. 
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A 
Grounded Normative Theory 


Genevieve Fuji Johnson 


Normative political theory, as a subdiscipline of political science, has evolved tremen- 
dously over the past four decades. Deliberative democratic scholars have been at 
the forefront of this evolution, propelling perhaps its most consequential turn—the 
turn towards the empirical through the use of interpretive, qualitative, and quanti- 
tative methodologies. In large part initiated by political theorist Jane Mansbridge in 
the 1980s, this turn was decisive; the course it set has been advanced and sustained 
by scholars including Simone Chambers (1996), John Dryzek (1990, 2000), James 
Fishkin (1991), Archon Fung (2006), John Gastil (2005), Carolyn Hendriks (2006), 
Ethan Leib and Baogang He (2006), Ricardo Mendonga (2008), Simon Niemeyer 
(2011), John Parkinson (2006), Boaventura de Sousa Santos (1998), and Graham 
Smith (2009), amongst others. Common to these theorists is a commitment to the 
rigorous engagement of the empirics of deliberative democracy. It is now standard 
practice for contemporary political theorists, working in a range of fields, to engage 
directly in the development, assessment, and analysis of forms of empirical evidence 
or empirical research materials, both of which are terms that include data from 
quantitative as well as interpretive and qualitative methodologies. 

Deliberative democrats may have set the course towards the contemporary union 
of empirical research and normative theorizing, but as the field of deliberative 
democracy has advanced, it has run the risk of co-optation by systems and insti- 
tutions of domination, including representative government, large corporations, 
and mainstream think tanks. It is not that these serve necessarily in domination; 
rather, it is that they appear stuck in status quo norms related to whiteness, mas- 
culinity, heterosexuality, ableism, and capitalism. As opposed to facilitating their 
democratization—as opposed to enabling the ‘civicization from below; to use James 
Tully’s words (2008, 4)—much of deliberative democracy seems only to further reify 
these systems and institutions. As Selen Ercan and John Dryzek point out, the success 
of the field may have been at the cost of its critical edge (2015, 244). 

Deliberative democracy’s initial Habermasian impulse was not only to examine 
practices, systems, and structures of domination, but also to advocate for real-world 
justice (see Habermas 1996; see also Dryzek 1990, 2000; Fung 2005; Fung and Wright 
2003; Hammond 2019; and Rostball 2008, 2009). An initial goal was to disrupt exist- 
ing democratic practices in order to prioritize inclusive and communicative over elite 
and strategic modes of engagement. Instead of disruption, however, many processes 
adopting the banner of deliberative democracy, and many studies of deliberative 
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democratic design, serve ultimately to reinforce domination. Their internal processes 
may meet the criteria of deliberative democracy—although this is not a given—but 
their outputs and outcomes often remain disconnected from contexts that persist 
in the oppression of, for example, Black and Indigenous peoples and other people 
of colour, gender non-binary/gender queer people, differently abled people, neuro- 
diverse people, undocumented im/migrants, hard drug users, inadequately housed 
people, people living in poverty, people who are highly stigmatized because of their 
work, and people living any combination of these identities. Given its close asso- 
ciation with status quo institutionalism, is deliberative democracy even relevant to 
contemporary struggles to achieve liberation from forms of oppression and justice 
for all? 

In this chapter, I offer insights from an emergent approach that not only blends 
empirical study with normative theorizing but does so recursively, inclusively, 
accountably, and even solidaristically, which may provide inspiration to deliberative 
democrats to develop their critical muscles. Identified by Brooke Ackerly and her 
colleagues (2021), Grounded Normative Theory (GNT)’ is premised on core com- 
mitments that may assist those wishing to pursue deliberative democracy in ways 
that challenge and combat forms of oppression and advance justice struggles. Upon 
outlining the basic contours of GNT as a broad field, I discuss how my understanding 
of ‘groundedness’ has evolved thanks to the writings of Leanne Betasamosake Simp- 
son and Glen Coulthard on grounded normativity (Coulthard 2014; Coulthard and 
Simpson 2016; Simpson 2015, 2017). Their work reveals pathways towards a firmly 
solidaristic approach to theorizing that is grounded in the voices and experiences of 
those who experience oppression. As I touch on in this chapter, in solidaristic theory 
and practice—for those of us who are settlers—grounded normativity and Indige- 
nous resurgence must be central in informing what solidarity is and what it entails. 
I then draw from my application of solidaristic engaged or grounded methodology 
and tentatively offer two suggestions to encourage deliberative democrats to conduct 
research in ways that are in solidarity with groups, communities, and nations seeking 
justice as they define it. 


The Basic Contours of Grounded Normative Theory 


Brooke Ackerly, along with Luis Cabrera, Fonna Forman, Chris Tenove, Antje 
Wiener, and myself (2021) unearth a body of scholarship constituting Grounded 


1 I came to the term ‘grounded’ from the grounded theory of Juliet Corbin and Anselm Strauss (2015), 
which is a dominant approach in qualitative methodology. I have since put time into learning about Glen 
Coulthard and Leanne Betasamosake Simpson’s much richer understanding of grounded normativity, 
which is rooted in place, territory, land, and water. Increasingly, in my own work, I’m moving away from 
the moniker of Grounded Normative Theory and toward something along the lines of Engaged Normative 
Theory. 
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Normative Theory. We identify the works of Mansbridge as precursor to this 
approach, which was both based on empirical research and in solidarity with ‘ordi- 
nary people. In her Beyond Adversary Democracy (1983), Mansbridge paid close 
attention to ordinary people and theorized on the basis of what they told her and what 
she observed about them. She writes that her experience of field work, which involved 
in-depth interviews and close observations, revealed quotidian ways in which ordi- 
nary people try to live by various ideals. How those with whom she spoke and whom 
she observed interpreted, navigated, and came to live by democratic ideals enabled 
her to broaden and clarify a range of normative questions—a range that would have 
otherwise not been available to her (Mansbridge 1980/1983, xiii). Her case studies of 
a traditional town in rural Vermont, ‘Selby; and of a participatory workplace crisis 
centre in an urban setting, “Helpline; enabled her to see, both practically and con- 
ceptually, gaps in democratic theory’s account of the importance of political equality 
for political legitimacy, how to address this gap by bridging adversary and unitary 
democracy, and ultimately how to realize certain aspects of community, comrade- 
ship, and selflessness without the need for all citizens to share common interests. It 
is important to point out, as Melissa Williams does, that Mansbridge was ‘among the 
first political scientists to highlight the fact that differences in rates of political par- 
ticipation across the lines of class, race, and gender can be traced in part to the way in 
which participation is institutionalized’ (2012, 798). In very thoughtful ways, there 
was a blend of scholarship and activism running through the work of Mansbridge. 
Recalling Mansbridge’s work of her early days on democracy but also on women’s 
rights reminds us that when we study deeply contested political issues, ‘analysis . . . is 
a political act, and one must, as a political being as well as scholar, take responsibility 
for iť (1986, xi, cited in Williams 2012, 798). 

Grounded Normative Theory as a field is, in large part, indebted to Mansbridge 
for doing the hard work of qualitative research involving everyday actors and, on 
this basis, normative theorizing about how they aspire to democratic ideals and 
about these ideals themselves. Many scholars have developed this type of approach, 
which is time- and energy-consuming. But, at its best, what we can identify as GNT 
accomplishes a fine blend of deep description, nuanced conceptual clarity, and a 
convincing normative orientation, and it points a way towards liberation and jus- 
tice in the real world (e.g. Ackerly 2018; Baines 2017; Cabrera 2020; Coulthard 
2014; Forman and Cruz 2018; Keating 2011; Tenove 2020). As Ackerly and her col- 
leagues observe, within contemporary political theory, there is now a widespread 
convergence on principles of comprehensiveness in research, recursivity between the 
empirical and normative, epistemological inclusion, and accountability to research 
partners, participants, and communities. We identify these as the core commitments 
of GNT. 

Ackerly and her colleagues write that comprehensiveness refers to the use by 
normative theorists of empirical methods—or empirically oriented methods— 
in order to enrich, widen, and deepen ‘the range of insights, claims, interests, 


Genevieve FujiJohnson 55 


and actors considered in their development of normative arguments’ (2021). 
Comprehensiveness refers primarily to the processes of collecting, assessing, and 
analysing research materials, and in these processes grounded normative theo- 
rists employ a wide-ranging set of tools from qualitative, quantitative, and mixed- 
methods approaches, including surveys, questionnaires, interviews, observations, 
experiments, focus groups, workshops, and document analyses. Comprehensive- 
ness does not necessarily map onto a linear research process, especially given the 
overarching recursive nature of the approach. 

Both comprehensiveness and recursivity may very well be the characteristics 
that unify the diverse field of GNT, including its non-solidaristic and solidaristic 
branches. Recursivity refers to the direct, immersive, and accountable incorporation 
of empirical analysis into normative theorizing. Like John Rawls’s ‘reflective equilib- 
rium, this process involves an analytical process of moving back and forth. In the 
case of Rawls, the theorist moves between considered judgements and general prin- 
ciples, revising both until they are mutually consistent. While Rawls’s approach is 
purely introspective, GNT scholars engage directly in primary empirical research, 
often immersing themselves in contextual details and relational dynamics character- 
istic of the field in which they are working. Moreover, they often forge relationships 
with and remain accountable to research partners and participants who share their 
insights and who contribute to empirical forms of evidence (Ackerly et al. 2021). 
This back and forth is more akin to a dialogue towards co-producing knowledge and 
co-developing normative claims through more formal research interviews and focus 
groups, or more informally through, for example, ongoing practical work, fundrais- 
ing, and strategizing. Thus, GNT is developed and refined through the ongoing and 
deep engagement with the empirics of the case(s). Where the research involves part- 
ners and participants, grounded normative theorists often continue engagement with 
them to collaboratively develop and refine the normative theory. 

Attentiveness to epistemological inclusion is also central for GNT scholars, espe- 
cially for those of us doing solidaristic research. Epistemological inclusion recognizes 
that some modes of inquiry silence certain voices and thus exclude forms of knowl- 
edge. It refers directly to ‘the ideas and direct insights, but also the ways of knowing 
and generating knowledge, of those engaged in political contestation’ (Ackerly et al. 
2021). This inclusion extends to claims to knowledge that are in ‘tension with the 
normative claims the theorist ultimately offers’ (Ackerly et al. 2021). This type of 
inclusion involves bringing such claims into analysis, figuring out what drives them, 
determining what is at stake in them, and working to either resolve or combat 
them within normative theorizing. If grounded normative—indeed, any normative— 
theorists are concerned with understanding and addressing various forms of oppres- 
sion, as they play out often in polarizing terms, epistemological inclusion must be 
a central methodological responsibility. Without this type of inclusion, the risk is 
too high that forms of oppression will remain obscured, but real, and will continue 
to take place. GNT scholars are often concerned with excavating unjust assumptions, 
practices, and institutions that can be hidden in plain sight; some of us work explicitly 
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to help address these injustices. For example, a number of GNT scholars seek to reveal 
deeper injustices obscured by dominant norms that are racist, sexist, homophobic, 
and ableist in nature. For those who take a more explicitly solidaristic orientation, 
the aim is to contribute to combatting these expressions of oppression. 

Epistemic accountability is another central methodological commitment in GNT. 
Primarily with respect to the interpretation of findings, this form of accountability 
entails managing the risk of misrepresenting a group or community’s desires, inter- 
ests, and needs when speaking and writing about its members (Ackerly et al. 2021). 
This form of accountability involves attending to potential power imbalances in the 
conduct of research itself and is particularly important in research either directly or 
indirectly involving participants. Even if research materials do not derive directly 
from research participants but instead from other sources, insofar as they are about 
a set of people or a community, GNT scholars typically exercise humility in trying 
to understand and express their views. They will often employ steps in interpretive 
analysis of what Peregrine Schwartz-Shea and Dvora Yanow call ‘member checking’ 
(2012, 106-107). Checking in with participants is a way of ensuring an adequate 
understanding of their perspectives as filtered through the interpretations of the 
researcher. But, especially for solidaristic GNT, this checking is a form of account- 
ability, allowing participants and researchers to work together to ensure that the 
normative theorizing not only makes sense to the participants but also aligns with 
their interests in advancing their justice struggles. Ackerly and her colleagues write 
that in more solidaristic approaches to GNT, ‘theorists foreground epistemic respon- 
sibility to ideas and persons disadvantaged in political struggles against exploitation, 
exclusion, oppression, and domination (2021). 


The Solidaristic Stance of GNT 


In my view, solidaristic Engaged or Grounded Normative Theory has tremendous 
potential to make contributions to groups, communities, and nations in their strug- 
gles for emancipation, justice, and resurgence (e.g. Ackerly 2008, 2018; Forman 2018; 
Forman and Cruz 2018; Johnson and Porth in progress; Sangtin Writers Collective 
and Richa Nagar 2006; Tenove 2015, 2020). Researchers have resources—including 
money, time, expertise, social status, public reach, etc.—that can be directed towards 
advancing struggles for justice and not simply studying such struggles. Underlying 
the solidaristic approach to GNT is a political solidarity—a solidarity that entails 
actively taking a side in struggles for justice with those experiencing oppression (e.g. 
Scholz 2008; Shelby 2002). As Sally Scholz writes, political solidarity ‘unites individu- 
als on their shared commitment to a political cause in the name of liberation or justice 
and in opposition to oppression or injustice’ (2007, 38). Solidaristic GNT is part of a 
larger tradition of scholarly activist work in which scholars deploy their resources in 
the service of activists. 
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Solidaristic GNT necessarily involves direct engagement with community mem- 
bers in learning about their justice struggles. It involves ongoing, genuine dialogue 
with those struggling for justice and empathetic scrutiny to assess and bolster their 
claims. It involves learning from and developing with them a vision of a future of lib- 
eration within a just social, economic, and political order and committing to working 
with them—as scholars and as activists—to realize this vision. Theorists engaging in 
this work commit to comprehensiveness, recursivity, inclusion, and accountability, 
but they focus explicitly on advancing the justice claims by people, communities, or 
nations experiencing oppression. They commit to the meaningful inclusion of voices 
and epistemologies of the oppressed and to accountability through dialogue with and 
action alongside them in their justice struggles. 

In my own work, I am learning from the writings of Indigenous resurgence 
scholars, and in particular the writing—and the spoken word—of Michi Saagiig Nish- 
naabeg scholar Leanne Betasamosake Simpson (e.g. 2015, 2017). From her blend 
of scholarly, artistic, and political expressions, my understanding of groundedness 
has radically shifted from one with roots in sociological grounded theory to one 
that is much more powerfully based in place—‘place’ as we might understand it in 
a very literal sense, but also place that is much more expansive, dynamic, and norm- 
generating. I am still learning about this understanding of place-based grounded 
normativity (Coulthard 2014), but as I understand it, it refers very directly to life, 
land, and water. Living within place involves recognizing our dependency on land 
and water and sharing necessities with others, present and future. There is a deep 
respect for the totality of place that generates knowledge about ways of living col- 
lectively and responsibly. Along with Yellowknives Dene scholar Glen Coulthard, 
Simpson develops their conception of grounded normativity, which takes norma- 
tive direction explicitly from groundedness in place (Coulthard and Simpson 2016). 
In their words, from groundedness, we can learn ‘how to be in respectful diplo- 
matic relationships with other Indigenous and non-Indigenous nations with whom 
we might share territorial responsibilities or common political or economic inter- 
ests’ (Coulthard and Simpson 2016, 254). Their articulation of grounded normativity 
is a powerful frame through which to understand the past and present violence 
of colonialism, as well as possibilities for Indigenous resurgence. For Simpson and 
Coulthard, it is from this understanding of groundedness that they practice their 
solidarity with others who are struggling against ‘white supremacy, anti-Blackness, 
heterosexual and cis-male dominance, and/or the violence of the state (Coulthard 
and Simpson 2016, 251). Similarly, Sarah Hunt, member of the Kwagiulth community 
of the Kwakwaka’wakw nation, and Cindy Holmes, a white settler living on the terri- 
tories of the Musqueam, Squamish, and Tsleil-Waututh people, emphasize building 
allyships and solidaristic connections based on relationships and relational knowl- 
edge production through conversation and storytelling, while also recognizing the 
immediacy of ongoing settler colonialism on the lands in which these relationships 
are forged (Hunt and Holmes 2015). 
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Drawing from these scholars, I understand solidarity with those who experience 
oppression to involve a recognition of the historical and ongoing dispossession of 
Indigenous peoples and a commitment to addressing this dispossession by work- 
ing to empower Indigenous peoples. It involves a commitment to advancing their 
causes and interests, as they lead the way in advancing them, and to being motivated 
not simply to feel good about ourselves and our efforts but to contribute construc- 
tively and purposefully towards their liberation and resurgence. As Simpson writes, 
it involves actively keeping in check any tendencies towards ‘saviourism, actively 
decentring whiteness, calling out white privilege and supremacy, and prioritizing 
Indigenous peoples as autonomous agents. It involves a genuine willingness ‘to join 
in the ways that Indigenous peoples are already organizing at the community level’ 
(Simpson 2016, 20). What I learn from these writings is that any expression of solidar- 
ity with those who experience oppression has to work in concert with Indigenous-led 
endeavours to end colonialism, return Indigenous lands, and support the resur- 
gence of Indigenous peoples. We should all ensure that, as Coulthard and Simpson 
write, ‘when we present our work on solidarity against the ‘misery of . . ., we are not 
standing on the backs of Indigenous peoples but instead engaged as related com- 
rades joined in critical co-resistance against the convergence of forces that divide 
and conquer us and the Earth on which we depend’ (2016, 250). This understanding 
of grounded normativity and political solidarity can help us—those of us who are 
interested in deploying our resources as researchers and scholars towards the ends 
of ending oppression and realizing liberation—deepen our thinking about how to 
contribute to movements towards racial justice and Indigenous resurgence. 


Contributions from Solidaristic GNT to Deliberative 
Democracy 


Nicole Curato, along with Dryzek, Ercan, Hendriks, and Niemeyer, suggests that 
deliberative democracy may be up to the task of addressing forms of racial injus- 
tice, claiming that its practice and theory are inclusive, involving multiple sorts 
of communication, and that it can be applied in deeply divided societies ‘to good 
effect’ in bridging ‘deep conflicts across religious, national, racial, and ethnic lines’ 
(2017, 34). But it is far from clear how deliberative democracy can contribute 
meaningfully to addressing racism that is deeply embedded in ideological, social, 
political, and economic systems, and that is bolstered by a dominant culture of white 
privilege and supremacy. It is even less clear how deliberative democracy can con- 
tribute to decolonization and Indigenous resurgence, particularly given its tethering 
to liberal institutionalism, its strong association with colonial states (e.g. citizens’ 
assemblies and citizen initiative reviews are directly linked to representative gov- 
ernment, intending to supplement or even bolster it) and extractive corporations 
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(e.g. deliberative polls have often been deployed by energy utilities, arguably to legit- 
imize them; Johnson, 2015). Despite being based on important ethical principles 
central to liberalism, deliberative democracy can easily serve as a ‘ritual’ of state 
power reinforcing colonial world views, practices, policies, and institutions (Ashforth 
1990). Moreover, Indigenous cultures have their own, very long-standing protocols 
for making collective decisions; the assumption that deliberative democracy can con- 
tribute to capacity building within Indigenous communities or that it has anything 
to add to Indigenous governance needs to be seriously interrogated. 

Drawing from my understanding of solidaristic GNT, there may be ways in 
which deliberative democratic practices and studies are implemented in solidarity 
with those experiencing racialized injustice and colonial violence. In terms of basic 
principles that may be helpful, I offer two. First, deliberative democrats need to 
acknowledge the ways in which their practices and studies may perpetuate forms of 
racialized and colonial marginalization and violence. They need to take stock of their 
own assumptions and biases to begin dismantling racist and colonial aspects of their 
work. Exercising intellectual humility, they need to build relationships with racialized 
and Indigenous communities in order to learn from them, to follow their lead, and 
to be accountable to them. This involves a serious commitment of time and energy 
and extends well beyond academic endeavours. Second, and closely related, delibera- 
tive democrats should be critical of highly structured micro forums, especially those 
connected to policy processes of states and corporations. Principles of solidaristic 
GNT may help to ensure that any re-designs function to resist entrenched forms of 
oppression. But, in addition, deliberative democratic scholars should continue and 
deepen their studies of macro forums—again, while exercising intellectual humil- 
ity as well as cultural awareness—especially grassroots organizing by community 
leaders, including Indigenous Elders, Hereditary Chiefs, Indigenous organizations, 
and their acknowledged allies. In the field of macro activities, there is much more 
openness for diverse forms of communicative exchanges, even those that are tempo- 
rally, spatially, and culturally different from mini-publics associated with deliberative 
democracy. Often contestatory and oppositional, macro activities can enhance and 
are necessary in truly public deliberation. For example, they can serve in organiz- 
ing and mobilizing those who experience oppression, revealing obscured forms of 
knowledge, provoking shifts in public opinion, and scrutinizing those in political 
power. 

One such study, in which I attempt to do this work, focuses on the grassroots 
organizing around missing and murdered Indigenous women and girls in Vancou- 
ver, British Columbia (2020). In the Canadian province of British Columbia (BC), 
Indigenous women and girls have gone missing from, and been found murdered 
in, northern BC, Vancouver's Downtown Eastside (DTES), and regions around the 
city. In response to these profound tragedies and to the prolonged inadequacy of 
responses by police departments, municipal governments, and the provincial govern- 
ment, family members, grassroots women’s organizations, Indigenous organizations, 
and Indigenous and other public leaders mobilized in an attempt to build pressure for 
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an official inquiry into this violence and for the implementation of strategies to put 
an end to it. For more than twenty years, they engaged in peaceful demonstrations 
that would evolve into political protest. These forms of organization and expression 
were eventually successful in bringing about an official inquiry in 2010.” Were it not 
for their efforts, it is very likely that the inquiry would not have occurred. But, from 
its outset, the BC Missing Women Commission of Inquiry was seriously flawed, and 
when this became clear, Indigenous women-led protests served to reveal the ways 
in which the inquiry replicated the same kind of exclusion and marginalization that 
contributed to the forsaking of so many missing and murdered women. 

My study of these demonstrations and protests, which was informed by principles 
of solidaristic of Engaged or Grounded Normative Theory, highlights the critical role 
played by Indigenous and women’s organizations in pushing for an official inquiry 
and in exposing its troubling flaws. Engaging in recursivity between my understand- 
ing of the empirics of the case and the normative claims I was identifying, tending 
to epistemological inclusion and accountability, I deepened my understanding of the 
crucial importance of listening to, and acting on the insights of, historically oppressed 
peoples, including many Indigenous women and non-Indigenous women living and 
working in the DTES. This can help to ensure that theories and practices of col- 
lective governance address the complexity and tragedy characterizing many social 
problems conditioned by deep structural inequalities and sexist, racist, and colonial- 
ist ideologies. Ultimately, this study revealed to me not merely that contributions of 
marginalized and experiential knowledges expressed in peaceful demonstration and 
protest can contribute to less oppressive and more emancipatory social and political 
systems, but also that they are necessary in this endeavour. Again, it is important to 
point out that this ‘study’ was much more than just that. It emerged from several years 
of volunteering in the DTES with women-serving organizations and involved ongo- 
ing discussions and relationships with community members and leaders. I remain 
committed to supporting members of this community. 


Challenges for Deliberative Democracy 


Several deliberative democratic scholars are engaging in work that may be under- 
stood in terms of GNT, orienting themselves to the empirical in order to expand their 
normative questions and claims; some appear to be doing so within a Mansbridge tra- 
dition of activist scholarship. In the work of scholars such as Ercan, Afsoun Afsahi, 
and Edwina Barvosa, we see attentiveness to how marginalized people participate 
in deliberative democratic processes. Afsahi explores how differently abled individ- 
uals and communities participate within deliberative systems (2020a). In another 


? It is important to note this inquiry investigated not only the disappearance and murder of Indigenous 
women but also of non-Indigenous women from the DTES during the 1990s and into the 2000s (see Oppal 
2012). 
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study, she draws from original survey and experimental data to observe and the- 
orize women’s willingness to deliberate (Afsahi 2020b). On the basis of this study, 
she develops two innovative approaches to better ensuring that women participate 
equally in deliberative forums (2020b). Similarly, motivated to advance the interests 
of women, Ercan (2017) explores how deliberative democracy—especially as dis- 
persed through a system of public discourse—has the potential for promoting a form 
of dialogue with violent members of illiberal migrant communities around the treat- 
ment of women. She moves between the theorizing of Dryzek, Monique Deveaux, 
and William Connelly and her case study work on ‘honour killings in the UK. She 
ultimately argues that there are conditions in which discursive contestation could 
occur between those advancing the rights of women and those who are violently 
opposed to the rights of women. 

With reference to the rights of members of LGBTQ2S+ communities, Barvosa 
engages in extensive empirical work—both qualitative and quantitative—to develop 
the claim that at least one large-scale deliberative system has emerged in the US in the 
form of a ‘publicly self-assembled, national-scale, public engagement’ on the social 
and legal equality of members of queer communities (2018). Barvosa’s empirical 
analysis and normative theory make clear that achieving the promise of delibera- 
tive democracy on a large scale requires not only the hard, concerted work of demos 
members towards social, procedural, and institutional change, but also the hard ‘self’ 
work of each to overcome her/his/their cognitive obstacles of implicit bias, iden- 
tity threats, and fear. Barvosa encourages deliberative democrats to research and 
theorize about not only fear and trauma, but also racialized, gendered, and het- 
eronormative privilege and its classic defensive manifestations of avoidance, denial, 
tears, and anger. A logical conclusion of her argument is that deliberative systems 
are possible only if we take on the responsibility to scrutinize our personal biases 
and to overcome cognitive obstacles and consider our fellow demos members as 
equally worthy of meaningful inclusion in democratic decision-making, no mat- 
ter their sexual orientation, gender expression, race, religion, ability, or citizenship. 
The works of Afsahi, Ercan, and Barvosa constitute an important contribution to 
ensuring that the field of deliberative democracy has a critical edge and contributes 
to real-world struggles against oppression. But for those of us who take seriously 
analysis as a political act and our responsibilities to it, there is more work to be 
done. 

In a recent article, Fung briefly touches on some related challenges facing deliber- 
ative democracy. As he writes, for ‘some social justice advocates, this moment calls 
for increasing conflict rather than focusing on public reasons and the common good’ 
(Fung 2020, 75). Fung goes on, stating that ‘after all of those who benefit from institu- 
tional racism, capitalism, sexism, and globalism have been cancelled from the polity, 
there might not be all that much polity left’ (2020, 75). Fung appears to be referring 
to the so-called ‘cancel culture, which is puzzling, perhaps even troubling. It sug- 
gests both a reduction of struggles—many of which are matters of life and death—to 
something akin to cancelling dinner plans with people we have recently decided we 
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do not like, and a need to retain people who benefit from oppression in the polity as 
if the polity depended on them. 

My suggestions, drawing from GNT in a solidaristic expression, are intended to 
encourage deliberative democrats not to succumb to the rhetoric of cancel culture, 
but instead to acknowledge their participation in oppressive practices and institu- 
tions and to listen to, learn from, and work in solidarity with those on the frontline 
of struggles against multiple and real forms of injustice. They are intended to encour- 
age those implementing and studying forms of deliberative democracy to deploy 
their considerable resources of money, time, energy, and social capital directly in 
the service of taking on structures of domination, ending forms of oppression, and 
empowering those who experience this oppression. If deliberative democracy is a rev- 
olutionary idea, if it has a critical edge, deliberative democrats would do well to step 
up their game by adopting principles of Engaged or Grounded Normative Theory, 
particularly those of recursivity, inclusivity, accountability, and even solidarity. 
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5 
Democratic Theorizing 


Hans Asenbaum 


In recent decades, deliberative democratic theory has developed an ambitious nor- 
mative programme, characterizing democracy in terms of inclusion, diversity, listen- 
ing, and transparency (Chambers 1996; Dryzek 2000; Young 2000). Many delibera- 
tive democrats, moreover, claim that democracy is not restricted to state institutions 
but extends to every sphere of human and even nonhuman interaction (Javier and 
Dryzek 2020). These ideals are generated in an unwelcoming context. Modern-day 
academia is characterized by hierarchical organization, competitiveness, and output 
orientation. For the most part, deliberative theory to date has been produced by a 
particular group of people who have undergone years of academic training in educa- 
tive institutions that shape the way we, as democratic theorists, think, speak, and act. 
This process not only produces a certain kind of thinking but is often accessible only 
to a few who are equipped with the necessary educational and economic resources. 

This chapter problematizes this exclusivity and offers a way of producing delib- 
erative democratic theory in accordance with deliberative ideals. The two questions 
this chapter seeks to answer are as follows: (1) How can deliberative democratic val- 
ues be realized in the process of generating democratic theory? and (2) How can 
those outside academia be included in deliberative theorizing? To answer these ques- 
tions, the chapter presents a novel approach. Instead of ‘theorizing democracy —the 
established approach to generating democratic theory through an engagement with 
academic literature—I advance democratic theorizing as an alternative (Asenbaum 
2022). Democratic theorizing theorizes democracy in a democratic manner. In this 
process, theorizing itself becomes an inclusive, diverse, and transparent deliberative 
process. 

The point of departure for building such an approach is the acknowledgement that 
we need a method of theorizing that allows theorists to leave their ‘ivory tower and 
engage directly with the life experiences of others. Grounded theory, especially in its 
recent renditions of critical grounded theory, offers a valuable starting point for this 
(Redman-MacLaren and Mills 2015). Here theory emerges out of an immersion in 
the field and a deep engagement with others’ life experiences. This approach can be 
deepened further by drawing on participatory research, a recently evolving approach 
(distinct from action research) that advocates the active engagement of those outside 
academia in the research process (Bussu et al. 2020). In addition to grounded theory 
and participatory research, a third resource for democratic theorizing can be found in 
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assemblage theory (Fox and Alldred 2015). This step allows us to include nonhumans, 
things, events, and natural forces in democratic theorizing. 

The approaches of grounded theory, participatory research, and assemblage theory 
are usually used for empirical research. Bringing them together in a novel approach 
to create normative theory presents new opportunities, but also significant chal- 
lenges. That this is a worthwhile endeavour is also claimed by Ackerly and colleagues 
(2021, 19) who recently argue that ‘normative political theory needs to be informed 
by empirical insights: In proposing to ground normative theory, they highlight the 
normative theorizing that deliberative democrats such as Mansbridge (1983) already 
do today (see Genevieve Fuji Johnson: Chapter 4 in this volume). In this chapter I seek 
to contribute to this agenda by proposing a step-by-step guide to the process of demo- 
cratic theorizing. My suggestion builds on the work of feminist and decolonial studies 
(Houh and Kalsem 2015), which have developed a rich repertoire of participatory 
techniques for ‘collective theorizing’ (Vaccaro 2020). 

The goal of democratic theorizing is to enhance the formative agency—the abil- 
ity to shape the norms we live by (Dryzek and Tanasoca 2021)—of those out- 
side academia. The resulting theory need not be ‘better’ than theory produced 
in the established manner. However, we can expect such theory to be closer 
to the various lifeworlds of humans and nonhumans outside ‘the ivory tower’ 
(Hendriks et al. 2020). 


Theorizing Deliberative Democracy Democratically? 


The field of deliberative democracy, and political science more generally, offers little 
guidance on how to theorize. While the texts on how to conduct empirical research 
are innumerable, theory is for the most part produced in a black box. Theories are 
the product of theorists’ theorizing. That’s it. 

Lately, however, some deliberative and democratic theorists have begun to open 
up the black box of theorizing. Three different approaches to theorizing democracy 
have emerged: First, the professional approach, which conceptualizes theorizing as 
restricted to the professional domain of academics. Second, what I call the ana- 
lytical approach, which engages with people outside academia as research subjects 
to include their viewpoints in theories of democracy. Third, the emphatic appeal, 
which fervently calls for including those outside academia in the theorizing process 
as co-theorists, but provides little guidance on how to achieve this. 

The professional approach to theorizing democracy can be exemplified by Mark 
Warrens problem-based approach. Warren (2017) presents a ‘strategy for construct- 
ing democratic theories’ (39) by highlighting the questions that democratic theorists 
need to ask. While this provides valuable guidance for (aspiring) democratic theo- 
rists, it clearly assigns theorizing democracy to a professional domain. In a similar 
vein, Fleuß and Schaal (2019) describe theorizing democracy as a problem-solving 
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activity that produces normative theory in close interchange with empirical reality. 
They raise the question: “What are we doing, when we are doing democratic theory?’ 
The ‘we’ in this question clearly refers to the academic community of professional 
theorists. 

The analytical approach, in contrast, seeks to enrich democratic theory with the 
viewpoints of people outside academia. Dryzek and Berejikian (1993), for example, 
directly address the problem of the exclusion of the demos from theorizing democ- 
racy. They employ Q methodology to survey people's understandings of democracy: 
‘We intend to develop a truly reconstructive approach that lets subjects speak for 
themselves about their interactive competences, and the categories (in our case, 
democratic theories) that these competences help construct’ (49). In a similar vein, I 
have engaged in theorizing democracy by interviewing activists of the Russian ‘infor- 
mal movement’ about their visions of democracy during the Perestroika period. The 
results provide vivid accounts of liberal, anarchist, socialist, and green theories of 
democracy (Asenbaum 2012). 

A third group of democratic theorists articulates emphatic appeals for a more 
active role for research participants in theorizing democracy. James Bohman (1999) 
criticizes critical theory which, despite its emancipatory ethos, understands the the- 
orist as enlightened intellectual, as ‘critic with a superior status over and above 
the limits of the participants’ perspectives’ (460). The hierarchical organization of 
science—which divides the roles of the researcher who controls the setting, and the 
subject whose reactions to manipulation are measured—is also reflected in norma- 
tive theorizing. In contrast, critical normative theory needs to embrace a deliberative 
ethos of cooperation: ‘critical social inquiry ... addresses the subjects of inquiry as 
equal reflective participants, as knowledgeable social agents. In this way, the asymme- 
tries of the context of technical control are suspended’ (474). Similarly, Hendriks et al. 
(2020) stress the benefits of incorporating the experiential lifeworlds of the demos 
into conceptions of deliberative democracy. They ‘call for a more grounded under- 
standing of connectivity in the deliberative systems thinking. The authors go on to 
argue that by studying ‘the way contemporary politics is experienced and enacted on 
the ground by citizens, politicians, and policymakers, deliberative democrats are bet- 
ter placed to identify a broader suit of practical, effective, and sustainable approaches 
to democratic repair’ (31-32). 

So, if we acknowledge that the inclusion of non-academics in the process of theo- 
rizing democracy is desirable because it enacts deliberative norms, how might we go 
about doing this? 


Grounding Deliberative Democratic Theory 


Developing an approach to theorizing that realizes deliberative norms is a challeng- 
ing undertaking, not least because of the context within which this process occurs. 
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Deliberative theory is produced in an academic context, one that is heavily shaped 
by professional hierarchies, a division of labour, competition, and output orientation. 
So how can the process of theorizing be democratized? 

Thankfully, there are some established methodological approaches that are expe- 
rienced in enhancing the role of non-academics in academic knowledge production. 
To develop democratic theorizing, I start from the grounded theory method (GTM). 
Instead of understanding theory as the product of an enlightened thinker who 
remains outside and above the object of research, GTM requires theorists to immerse 
themselves in the field and create theory inductively. The result is theory that reflects 
not only the perspective of the theorist, but also those of many participants: ‘By start- 
ing with the data from the lived experience of the research participants, the researcher 
can, from the beginning, attend to how they construct their worlds. That lived expe- 
rience shapes the researcher’s approach to data collection and analysis’ (Charmaz 
1990, 1162). Despite this promising outlook, to my knowledge there is no demo- 
cratic theory produced through GTM. And despite the fact that GTM is the most 
used methodological approach in the social sciences, it has hardly been applied in 
political science (Becker 2012). 

While there are different versions of GTM (Timonen et al. 2018), a few basic steps 
are common to all of them. Grounded theorists approach the field with an open mind. 
While the original version of GTM urged theorists to forgo an engagement with the 
existing literature on a given topic (Glaser and Strauss 1967), today a more flexi- 
ble approach is advocated that allows for, and indeed encourages, a back and forth 
between fieldwork and literature engagement (Thornberg 2012). Research questions 
are either constructed before approaching the field, in which case they are supposed 
to be open rather than hypothesis-driven, or they emerge from the engagement with 
the field (O'Neil Green et al. 2007, 477). The core of conducting GTM consists of 
a constant move between collecting and analysing data. While qualitative methods 
such as textual analysis, semi-structured interviews, focus groups, and observations 
are the most common choices, a mixed methods design, including quantitative meth- 
ods, is also frequently employed. The inductive data analysis begins with open coding. 
As patterns and core categories emerge through constant comparison of insights, 
both data gathering and coding become more selective. Through the continuous 
back and forth between data gathering, data analysis, and in some cases literature 
engagement, basic categories merge into categories of higher abstraction until a the- 
ory emerges that has made its way from a specific case to a general validity for other 
cases (Reichertz 2007). 

It is important to note that classical GTM was developed to produce empirical the- 
ory. In contrast, democratic theorizing aims at generating normative theory. If we 
are interested in the normative theories that GTM can produce, we have to also ask 
about its ontological underpinnings. In the original iteration of GTM by Glaser and 
Strauss (1967), this question was disregarded. GTM was seen as a normatively neu- 
tral approach, a tool to analyse reality and formulate generalizable principles. Later, 
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this version of GTM was identified as positivist and challenged by a constructivist 
version (Charmaz 1990). Rather than understanding reality as a stable condition 
external to the researcher, reality is co-produced through the theorizing process itself. 
This also fundamentally changes the roles of grounded theorists and the field they 
engage with. Interviewees are now seen as research participants who co-construct 
theory, rather than research subjects whose knowledge is extracted. More recently, 
GTM has benefitted from a third ontological perspective. Building on feminist and 
decolonial approaches (Mignolo 2021; Bhambra 2007), critical theorists have put 
forward what has been termed critical, transformative, and indigenous grounded 
theory, which harbours particular potential for democratic theorizing, as ‘critical 
indigenous grounded theory inquiry necessarily becomes disruptive . . . and radically 
democratic’ (Denzin 2007, 460). 

Critical grounded theory (CGT) and deliberative democracy share their norma- 
tive roots in critical theory. CGT builds on the emancipatory objectives of critical 
theory from the start of the theorizing process: ‘In CGT, the choice of research prob- 
lem is explicitly driven by moral and/or social concerns in an ambition to produce 
critical knowledge to enable social emancipation (Belfrage and Hauf 2017, 9). At the 
end of this process stands ‘theory that can be used to challenge excluding and oppres- 
sive structures and systems for positive change’ (Redman-MacLaren and Mills 2015, 
4). Although ‘[g]rounded theory, because of its commitment to critical, open-ended 
inquiry, can be a decolonizing tool for indigenous and non-indigenous scholars alike’ 
(Denzin 2007, 456), the role of those external to the research participants’ reality 
needs special attention. Critical grounded theorists who see themselves as allies of 
marginalized groups need to make a special effort to reflect on their class, race, and 
gender positionality, their assumptions and preconceptions, and to learn as much as 
possible about the lifeworlds of their participants (O’Neil Green et al. 2007). 

This still leaves open one central question: To what extent can critical grounded 
theorizing be employed to generate not empirical but normative theory? By moving 
away from its positivist origins and embracing the normative role of researchers, CGT 
naturally lends itself to normative theorizing. Genevieve Fuji Johnson makes a pow- 
erful argument that empirical methods can and should play a key role in developing 
normative theory (see Chapter 4 in this volume). In theorizing with those who strug- 
gle, grounded normative theory puts forward a broad agenda of inclusive, empirically 
driven theory. Democratic theorizing builds on this approach and aims to outline the 
concrete steps that such a research endeavour might entail. 


Theorizing as a Deliberative Process 


So far, we have established that, from a viewpoint of deliberative norms, theorizing 
democracy in a democratic manner is desirable and CGT provides a valuable start- 
ing point for this endeavour. What we are still missing is an understanding of how 
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those outside academia can be included in democratic theorizing. Here, insights from 
participatory research (PR) can help (Bussu et al. 2020). Over the years, PR has built 
considerable expertise in uncovering the deliberative potential of research partici- 
pants: ‘Participatory research methods are geared towards planning and conducting 
the research process with those people whose life-world and meaningful actions are 
under study’ (Bergold and Thomas 2012, 192). Growing out of action research, PR 
has only recently developed its own distinctive features. 

In conducting PR, researchers need to shift their roles from neutral investigators 
who analyse the world from a detached position, to engaged facilitators and enablers 
who provide participatory infrastructures (Cornwall and Jewkes 1995). Researchers 
need to engage with participants with genuine curiosity and humility as learners. “The 
aim of participatory or emancipatory research is to confer control over the “telling” 
and ownership of the data on to participants, and to give them opportunities to 
present something of themselves’ (Aldridge 2017, 28). In a sense, this turns the estab- 
lished academic hierarchies on their head. Researchers do not know better because 
of the number of books they have read and written. Rather, participants are always 
better informed about their personal life experiences; they are experts on their lives. 
This does not deny a special role of academic researchers who have different stakes 
and greater responsibilities. The decision of whether to apply PR methods is not a 
simple yes or no choice; rather, it is a matter of degree (Cornwall and Jewkes 1995; 
Aldridge 2017). The degree of the fulfilment of deliberative values within a research 
project also depends on the resources, energies, and willingness of participants. 

The deliberative norms of democratic theorizing can further be expanded by 
paying attention to nonhumans in the theorizing process. Deliberative democratic 
theory increasingly asks how deliberation can include nonhuman animals (Parry 
2016) and natural events (Javier and Dryzek 2020). Indeed, even inanimate objects 
play an important role in deliberation (Bennett 2010, 94-109). What does this mean 
for a deliberative understanding of democratic theorizing? Lupton (2018) proposes 
to understand personal data generated through digital devices as a lively compan- 
ion species. They are born through human and machine interaction, and they are 
continuously interpreted differently. In a similar vein, Schmidt (2019, 148) under- 
stands matter as a participant in social practices: ‘things and non-humans appear 
as participants: each collaborates in its specific materiality and is involved in its 
practice-specific manner. 

Including data and other nonhumans in the research process as participants leads 
us to an understanding of what Fox and Alldred (2015) call the ‘research assem- 
blage. Research assembles researchers with their personal histories, intentions, and 
interests, research methodologies, theories, literature, data, software, computers, 
laboratories, (home) offices, and libraries. Assigning data, methods, theories, and 
software a more active role means raising questions such as: ‘what is this data doing 
in relation to this inquiry? (Timonen et al. 2018, 7). Theorists need to be atten- 
tive to how the use of a certain software might structure their thinking, how the 
confrontation with certain data makes them feel, and how the research project affects 
their personal life. Not only the research process, but the phenomenon under study 
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consists of an assemblage. Understanding the interaction between academic and 
non-academic theorists as the interface between a hybrid research-event assemblage, 
enhances sensitivity to the interaction of the various parts involved. 

CGT, PR, and assemblage theory all stress the value of method and data trian- 
gulation (Fox and Alldred 2015, 408), well expressed in the classical GTM credo 
‘all is data. This also resonates with recent developments in deliberative democracy 
research, which call for greater attention to be paid to nonverbal expressions, such 
as images, sounds, and presence (Curato 2019; Mendonça et al. 2020). Assemblage 
thinking lends further credit to this approach. It expands the methodological reper- 
toire of original GTM from interviews and document analysis to including visuals, 
artistic performances, and material artefacts. Visual grounded theory, for example, 
has potential for democratic inclusion, as ‘marginalised individuals or communities 
are able to construct visual representations of how they personally experience and 
understand their lives’ (Liebenberg et al. 2012, 60). 


Democratic Theorizing in Practice 


Combining the key insights from CGT, PR, and assemblage theory, I propose demo- 
cratic theorizing as a method for developing theory in a deliberative manner. In what 
follows, I will describe the concrete steps that researchers, wishing to employ this 
method, might undertake. In doing so, I am guided by my own experience in con- 
ducting a democratic theorizing project together with Black Lives Matter activists. 
Not all the steps elaborated below need to be taken, nor do they need to be done 
in this order. Each individual project will differ depending on the specific research 
topic, context, and personal preferences. 

Phase I: Ideation: The democratic theorizing process starts with developing a plan. 
We can think of it in terms of a research design that guides the theorizing endeav- 
our (Wahyuni 2012). This plan may be contingent, up for revision, and may change 
through the research process. Let’s think of this kind of planning more in terms of 
ideation—collecting ideas about how to bring together the research assemblage (Fox 
and Alldred 2015). Democratic theorizing entails a sensitivity to phenomena, actors, 
or topics at the periphery or outside of mainstream discourses. These actors can play 
a crucial role in the ideation phase. If they are nonhumans, there may be ways of 
listening to them, however they may express themselves. Ask yourself: What connec- 
tions can I establish with ‘the field’ early on? Get in contact with some actors related 
to your topic and ask them for advice. It might also make sense to invite academic 
or non-academic collaborators and form a team. While non-academics might not 
have the time or other resources necessary to be involved full-time, offer them the 
chance to contribute to the project on their terms. Think about what resources are 
available to you for offering financial or other types of compensation to participants. 
Start forming a network around the theorizing project. You may also advertise your 
idea on social media or on a public project website and ask who would like to 
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contribute. For my project, I set up the Democratic Theorizing Project website to 
call for participation (Asenbaum 2021). 

Once you have your team or network in place, there are many choices to be 
made: What are the central research questions? Which methods are best suited to 
answer these questions? Keeping in mind that democratic theorizing favours a multi- 
methods approach, which methods complement each other? Who should be invited 
to participate? Who will play which roles in the theorizing process? This first phase 
is also an important time for self-reflection. Why are you drawn to this topic? How 
does it relate to your personal life and biography? What power asymmetries between 
you and research participants may be at work? 

Phase II: Diving in at the deep end: We can never simply forget all the prior knowl- 
edge or opinions we have about the subject we are researching. In fact, if we are drawn 
to a particular topic, we may already know quite a lot about it. This is not a problem. 
Democratic theorizing, however, works best with an open mind. To truly listen, it 
is better to not block our receptive capacities with the academic knowledge that is 
already out there. This is why, unlike in traditional research designs, the literature 
review in democratic theorizing comes last. After the first phase of ideation, data 
generation and analysis start immediately. Diving in at the deep end may be a little 
scary, but it is also fun! This can take many forms: photo elicitation, interviews, social 
media analysis, surveys or more engaged forms such as the Community of Inquiry 
or the Deliberative Camp (as introduced in Chapters 31 and 32 in this volume). 

For my project, this phase focused on social media analysis. On Twitter and Insta- 
gram, I collected 323 social media posts with the hashtag #BlackLivesMatter. The 
contrast between the textuality of Twitter and the visuality of Instagram played an 
important role in creating a vivid research assemblage. Images spoke a different lan- 
guage and conveyed affectivity and emotion. I coded each post with the help of a 
qualitative research software, before collecting the next. This allowed me to take the 
insights from each bit of analysis to the next. Rather than gathering a big pile of data 
and then coding them all at once, the recursive movement back and forth between 
collecting and analysing allows an understanding to emerge gradually and organi- 
cally (Reichertz 2007). Think of this process not so much in terms of analysis but 
more as a conversation with data. How does the data respond to your questions and 
how do you react to the data? I also started writing memos—short notes to record any 
thoughts and insights that came up during the coding process (Bex Lempert 2007). 
These can be an important source for writing up the theory later. 

Phase III: Listening: The next phase focuses on active listening and an open 
exchange. I entered into conversations with twelve Black Lives Matter activists 
in online interviews.’ To invite participants as widely as possible, I posted open 
invitations on social media and set up a public website that invited anyone to con- 
tribute to the project (Asenbaum 2021). Throughout the entire theorizing process, 
I published current findings on the website and shared them on social media, to 


1 This project has been approved by the Human Resource Ethics Committee of the University of 
Canberra, ID 4746. 
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enhance the deliberative value of transparency. I also contacted various groups in 
the movement directly. 

The online conversations with activists were video recorded, so that not only their 
words, but also their body language could feed into the interpretation. In the con- 
versations, I asked big questions, such as “What does “life” mean for you?’ It was 
fascinating to hear their brilliant responses. Rather than traditional interviews, these 
were philosophical conversations. Everyone is a philosopher. Yet, most people do not 
know that they are one. Giving an interview can sound quite intimidating. When I 
invited one activist to an interview, he said: ‘Let's just talk. Pm not good at interviews: 
Accordingly, I also shared my own opinion in these conversations or referenced pre- 
vious conversations with other activists. This made conversation partners feel more 
at ease. The process provided several opportunities for participant feedback. During 
the interviews, I summarized my interpretations of what participants had said, to give 
them an opportunity to correct me. After each interview, I sent the complete inter- 
view transcript to participants along with a one-page summary in which I highlighted 
my main takeaways. I then asked for corrections and additions. 

I also conducted seven one-on-one participatory analysis sessions. I prepared a 
distinct set of social media posts for each session, which participants then interpreted 
in conversation with me. Knowing the movement from the inside, they often high- 
lighted things I was not aware of. I then fed their interpretations into the coding 
process and revised my original interpretations. 

The response to the public website and social media invitation I sent out at 
the beginning of this phase was tremendous and unexpected. People from various 
domains, including democratic practitioners, public servants, filmmakers, commu- 
nity organizers, and graduate students responded to the invitation. I held a series of 
conversations with them, both sharing my experiences of democratic theorizing and 
at the same time feeding their insights into the process. 

Phase IV: Immersion: An important element of democratic theorizing is to 
immerse oneself in the given topic by assembling multiple sources, inputs, and media. 
This can also take the form of temporarily infusing one’s own personal life with 
research content. How far one goes in this direction is a personal choice in which 
one’s own well-being and self-care need to be taken into consideration. 

During this phase, I made checking the social media hashtag #BlackLivesMatter 
a daily habit, which meant waking up and going to bed to Black Lives Matter con- 
tent. I constantly took screenshots and fed current content into the analysis process. 
I further enriched the process with non-academic sources regarding the movement, 
including journalistic and activist texts and YouTube documentaries. Working from 
home during the COVID-19 pandemic, I hardly had a meal without watching a Black 
Lives Matter documentary. My evenings were filled with episodes from the Netflix 
show Dear White People, which critically reflects on issues of racial identities and dis- 
crimination in the US. I kept a personal diary in which I documented insights and 
personal thoughts on each of these items. The diary was important for self-reflection 
about my role in the project and my interactions with participants and data. 
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Phase V: A stopover in the ivory tower: After having assembled different insights and 
experiences, we can now turn to the academic literature. A literature review helps to 
support or contrast findings, and to see what is new and what has been said before. It 
might be a little scary to do this last. What if what we have found is already known? In 
my experience, however, every project is unique. It requires time, work, and creativity 
to identify what is special. The established literature is a valuable source for enriching 
and deepening your normative theory. 

Phase VI: Bringing it all together: So where does the actual theory come from? Or 
better, how do all these varied things we have assembled translate into a coherent 
theory on paper? From the inductive grounded theory engagement, core categories 
will emerge. They build the core structure of the theory. Memos written throughout 
the entire process are an important source for writing up the theory. You may want to 
lay out categories on pieces of paper on the floor, write them on sticky notes attached 
to the wall, draw a mind map or visualize them through software graphs, all of which 
can be done alone or with participants. 

For me, after the social media analysis, conversations with various actors, partic- 
ipatory analysis sessions, multimedia input, personal diary, and literature review, 
things started to come together. I wrote a first paper draft. I sent this draft to the 
participants and asked for feedback. I then invited them to an online focus group, 
where we discussed the core categories of the theory and reflected on the process. I 
fed their reflections into the next iteration of the paper. 


Conclusion 


Democratic theorizing as introduced in this chapter aspires to realize the delib- 
erative ideals of inclusion, diversity, listening, and transparency in the theorizing 
process. In contrast to established approaches to theorizing democracy, which are 
limited to the professional domain of academic theorists, democratic theorizing 
aims at realizing the formative agency of people outside academia (Dryzek and 
Tanasoca 2021). It constitutes a common process of co-constructing theory with 
those who are the actual experts when it comes to their own lifeworlds. This 
kind of theorizing goes against the established logic of academia, which is situ- 
ated in a context of neoliberal market economies and governance structures. It 
requires theorists to step out of established thinking and knowledge production 
patterns. 

As a white, male academic from Europe, engaging with a Black-led US movement 
entailed some tensions, moments of suspicion, feelings of guilt (Luttrell 2019). Being 
a distant supporter but not an active participant in the movement sometimes made it 
hard to establish trust between myself and research participants. Perceiving myself as 
a reflective and progressive thinker and supporter of the movement, this process has 
shattered parts of how I see myself, my work, and the world around me. Acknowledg- 
ing my own privilege was painful at times, but it also allowed me to grow personally 
and as an academic. Growing pain is good. As democratic theorists, we should be 
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open to learning something new, not only about our research topic but also about 
ourselves. 

The guidance to democratic theorizing I have provided here is just one way of 
going about this. Democratic theorizing has the potential to fulfil the deliberative 
aspiration of diversity and this can take various forms. I hope that this approach will 
inspire future theorizing projects and contribute to a bigger movement towards a 
more democratic way of creating democratic theory. 
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PART Il 
MEASURING DELIBERATION 


6 
Discourse Quality Index 


André Bachtiger, Marlène Gerber, and Eléonore Fournier-Tombs 


How deliberative are communicative interactions in politics and society? This is one 
of the big issues for deliberative theorists and empirical scholars alike. This chapter 
focuses on the Discourse Quality Index (DQI), which provides a quantitative content 
analysis of deliberative quality. The DQI can be employed in a variety of settings. It 
has allowed researchers to examine the deliberative qualities and consequences of 
parliamentary interactions (Steiner et al. 2004; Bachtiger and Hangartner 2010; Lord 
and Tamvaki 2013) as well as deliberative mini-publics (Caluwaerts and Reuchamps 
2014; Gerber et al. 2018). It has also been used in experimental research (Baccaro et al. 
2016) and served as a template in the Varieties of Democracy project for assessing the 
deliberative aspects of a country’s politics (Coppedge et al. 2011). We document the 
DQI’s evolution, its blind spots as well as recent developments towards a machine- 
learning DQI. 


The Original DQI 


The original DQI was developed for the analysis of parliamentary debates. It is 
strongly rooted in a Habermasian-inspired understanding of deliberation, which 
emphasizes rational argumentation. In its original conceptualization, the DQI 
employs five indicators of deliberative quality (Steenbergen et al. 2003): 


(1) Participation Equality: When focusing on legislatures, participation is natu- 
rally restricted to MPs, so the DQI operationalizes participation in terms of 
interruptions (other than normal limitations on debate time) that makes it 
impossible for an MP to express his or her views. 

(2) Level of Justification: In politics (but also elsewhere), it is usually difficult to 
apply external and substantive standards to what constitutes a good reason. 
Hence, the original DQI focuses only on the syntactic structure of argu- 
ment and judges to what extent a speaker gives complete justifications and 
thus makes his speech accessible to rational critique. At the centre of the 
concept of justification is the inference in which the conclusion of content 
is derived from the premises (i.e. primarily reasons). The relations (links) 
between premises and conclusion may contain argument connectives such 
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(3) 


(4) 


(5) 


as ‘since; for, ‘so, ‘therefore, ‘because. Yet argument connectives can also 
be made implicitly. Consequently, coders cannot rely upon a predetermined 
list of connectives, but they must interpret. The DQI distinguishes between 
four levels of justification rationality: no justification; inferior justifications 
where the linkage between reasons and conclusion is tenuous; qualified jus- 
tifications where ‘decent rationales’ (Neblo 2007) are offered and a linkage 
between reasons and conclusion is made; and sophisticated justifications 
where ‘sophisticated rationales’ (Neblo 2007) are offered and a problem is 
examined in depth or from different viewpoints. 

Content of Justification: Drawing from Cohen (1989; and see Chambers 
2018), many deliberative democrats emphasize that arguments should be for- 
mulated with an eye to what we have in common and what is universal. The 
DQI measures whether arguments are cast in terms of narrow group or con- 
stituency interests, whether there is neutral reference or mixed reference (i.e. 
reference to both narrow group interest and common good), or whether there 
is a reference to the common good. With regard to the common good, the 
DQI focuses both on the common good stated in utilitarian terms, that is, 
as the best solution for the greatest number of people, and the common good 
expressed through the difference principle, that is, the common good is served 
when the least advantaged in a society are helped. 

Respect: According to Gutmann and Thompson (1996, 79), respect is one 
of the core principles of deliberative democracy: ‘it requires a favorable atti- 
tude toward, and a constructive interaction with, the persons with whom we 
disagree. The DQI measures respect with three dimensions: respect towards 
groups, demands, and counterarguments. The DQI respect indicators are 
based on the concept of ‘appraisal respect; that is, participants are expected to 
have (or to develop) a positive attitude towards other participants’ demands, 
arguments, and needs. Having a positive appreciation for the reasonableness 
of one’s political opponents may not mean that the content of one’s own posi- 
tions should be corrected, but it could mean that one should be willing to forge 
a moral compromise with them. The codes for the three respect dimensions 
involve disrespect, neutral evaluation, and explicit respect and agreement. 
Constructive Politics: This indicator is based upon the principal goal of classic 
deliberation to reach consensus. The DQI distinguishes between four levels of 
constructivity: positional politics; alternative proposals, that is, proposals that 
attempt to mediate but that do not fit the current agenda; consensus appeals, 
where speakers make an unspecific appeal for compromise or consensus; and 
mediating proposals that attempt to mediate within the same agenda. 


The goal of the original DQI was to produce a measurement tool that satisfies the 
(demanding) requirements of empirical scholars in various fields of the social sci- 
ences to create good instruments (e.g. Munck and Verkuilen 2002). In a series of 
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tests, it was demonstrated that the DQI has good-to-excellent inter-coder reliability, 
that is, there is generally broad agreement where a particular speech act falls on its 
indicators. We have much more to say on the aggregation of the various DQI items, 
but we first turn to conceptual blind spots that have led to several amendments of the 
original DQI. 


Blind Spots and Amendments: Updating the 
Original DQI 


The original DQI has blind spots on the measurement of participation equality, 
interactivity and reflection, alternative forms of communication, substantive argu- 
mentative quality and sincerity, as well as an appreciation of the sequential character 
of deliberative quality (and “deliberative moments”. There are further and more fun- 
damental blind spots of the DQI, especially context- and goal-sensitivity, to which 
we turn in the last section. 

Participation equality. Measuring communicative interruption does not tap into 
the dynamics of inclusion and equality in deliberation (Thompson 2008). Instead, 
as Dutwin (2003) has proposed, one standard for evaluating equal participation in 
deliberative processes is to focus on the frequency and the speaking time by specific 
social groups (e.g. gender or cultural minorities). Consequently, an updated version 
of the DQI follows Dutwin (2003) and Stromer-Galley (2007) and proxies equality (or 
domination) by counting the frequency and volume of participation, the latter mea- 
sured by the number of words. The latest development is Gerber’s (2015) distinction 
between ‘equality of participation’ and ‘equality of consideration. While equality of 
participation captures the frequency of participation, equality of consideration inves- 
tigates whether discussants are recognized as equal discussants with legitimate claims 
(Knight and Johnson 1997). To capture equality of consideration, Gerber (2015) anal- 
yses whether the arguments of specific social, cultural, and political groups receive a 
response from another group. The data obtained by this analysis also lends itself to 
network analysis; for instance, it can be used to check for ‘homophily’ in deliberative 
settings (Gerber 2015). 

Interactivity and reflection. The original DQI does not capture interactivity or 
reflexivity, or it does so only indirectly. With regard to interactivity, the original 
DQI tries to measure this in the respect dimension, namely under the rubric of 
counterarguments: it counts whether counterarguments are included or ignored. But 
this measure has not proved to be ‘sharp’ enough to thoroughly explore patterns of 
reciprocity and listening in deliberation. Consequently, Pedrini et al. (2013) have 
developed a new indicator for interactivity, assessing whether participants make a 
direct reference to other participants as well as to other participants’ arguments. 
Himmelroos and Christensen (2013), in turn, have made an effort to incorporate the 
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concept of reflection into the DQI. Concretely, they measure whether a participant 
considers a counterargument in their own argumentation or compares or weighs 
different arguments. Surely, both attempts are only approximations of the concepts 
of interactivity, listening, and reflection. A full-blown analysis would require that one 
also explores the ‘deliberation within’ component, investigating whether participants 
have listened to others but do not explicitly report this. 

Alternative forms of communication. Difference democrats have long argued that 
a focus on rational argument may be exclusive of specific social groups and speak- 
ing styles (Sanders 1997; Young 2000). Since a key goal of deliberation is to include 
‘all affected interests, a tension between inclusion and deliberative ability may arise, 
turning deliberation into a potentially harmful intervention that further marginalizes 
already disenfranchised groups. Both Sanders and Young stress the importance of 
alternative forms of communication—such as testimony, storytelling, or rhetoric—to 
avoid these constraints. Updated variants of the DQI have made an effort to incor- 
porate ‘storytelling’ and narratives. A new measure was created to check whether 
participants use personal narratives or experiences. 

Substantive argumentative quality and sincerity. While the DQI captures the formal 
properties of arguments, it says nothing as to the substantive quality of arguments, 
or the sincerity with which they are offered (Bachtiger et al. 2010; Neblo et al. 2018). 
Assume that we find an actor who scores highly on all DQI indicators, that is, pro- 
vides extensive and complex justifications for her positions, is oriented towards the 
common good, shows respect for other demands and counterarguments as well as 
empathy for the needy, and offers a mediating proposal of how to resolve the dispute. 
While this looks like an indication of true deliberative action, one cannot exclude the 
possibility that this actor is engaging in sophisticated cheap talk designed to manip- 
ulate the terms and outcomes of discussion (Schimmelfennig 2001). Only if we could 
see through one’s intentions, finding that one really means what one says, would 
we be in a position to make a case that this is true deliberative action. Even when 
additional evaluation criteria are applied—for example, no horse-trading, no side 
payments, opinion shifts in direction of positions of less powerful actors, or a high 
amount of normative arguments during discussion (Deitelhoff 2006)—it is difficult, 
yet not impossible, to make a statement regarding sincerity. 

Neblo et al. (2018) have supplemented the DQI with ‘substantive quality’ and ‘sin- 
cerity’ codes. They employed an ideologically diverse set of coders to check whether 
an argument was substantively sound and whether the argument was offered in sin- 
cere ways. Intercoder reliability levels on these categories were high (hovering around 
80 per cent); as Neblo et al. (2018) note, research in psychology suggests that peo- 
ple are generally apt at assessing a speaker's sincerity. We think that such codes for 
‘sincerity’ should be routinely built into assessments of deliberative quality. 

Sequential Character of deliberative quality and deliberative moments. Traditional 
DQI analyses have analysed entire debates based on the aggregation of individual 
speeches. But this ignores the fact that entire debates may not have a high deliberative 
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quality throughout; rather, deliberative quality may vary across discussion phases, 
requiring the adoption of a sequential perspective. Focusing on small group discus- 
sions, Gastil (1993, 123) noted that groups achieve high-quality deliberation only 
‘during brief, brilliant flashes’ that ‘last a few minutes, or perhaps an hour’ This 
requires that deliberative quality is analysed in sequences and ‘deliberative moments’ 
are identified in which various indicators of deliberative quality—justification ratio- 
nality, respect, constructivity—all score highly, whereas a focus on the entire debate 
would lead to a more ‘nuanced’ evaluation of deliberative quality (see Bachtiger et al. 
2010). 

The importance of focusing on ‘deliberative moments is also picked up by Steiner 
et al. (2017). They correctly claim that ‘existing instruments to measure the qual- 
ity of deliberation are too static, focusing too much on the analysis of individual 
speech acts. Instead, they propose to focus on so-called ‘transformative delibera- 
tive moments, where the level of deliberation is either lifted from low to high or 
drops from high to low. According to Jaramillo and Steiner (2014), one has to under- 
stand the dynamic and the context of a discussion to identify the situations in which 
the discourse becomes significantly lower or significantly higher from a delibera- 
tive perspective. This is accomplished in an interpretive way by coders. Sequencing 
deliberative quality is a first step towards a systemic approach that understands delib- 
eration as an ‘emergent’ phenomenon (Bachtiger and Parkinson 2019). It also taps 
into the complexities of aggregating deliberative quality, a topic which we will address 
below. 


Aggregation 


Since deliberation contains multiple attributes and relevant indicators, it must wres- 
tle with the aggregation problem, namely deciding which attributes and indicators to 
combine into a single index, whether to add or multiply them, and how much weight 
each component should be given. There also needs to be a correspondence between 
theory and the selected aggregation rule (Munck and Verkuilen 2002). For instance, 
if theory indicates that two attributes have the same value, one would simply add 
the scores; if they are necessary features, one could multiply both scores; or if they 
are sufficient features, one could take the score of the highest attribute (Munck and 
Verkuilen 2002). Last but not least, in order to draw conclusions about the aggre- 
gated deliberative quality, analysts also need to deal with the question of whether the 
attained scores are ‘good enough from a theoretical point of view, that is, they are 
required to set certain threshold values for high and low deliberative quality. We will 
address these points in turn. 

Simultaneity is a traditional expectation in aggregation, that is, good delibera- 
tors should ideally comply with all quality standards, and equivalence, that is, no 
priority or differential weight is given to specific indicators of deliberative quality 


88 Discourse Quality Index 


(see Gerber et al. 2018). Goodin (2005) calls this the ‘unitary deliberator model 
where all deliberative virtues are simultaneously and continuously on display. The 
DQI epitomizes these ideas and operationalizes deliberative quality as a unidimen- 
sional construct. For instance, the more justification in a speech, the more respectful 
it should be, and so on. If unidimensionality exists, then the aggregation of the var- 
ious components is easy (they are just counted together) and researchers obtain 
a powerful measurement instrument yielding results that can be interpreted in a 
straightforward way (the higher the score on the scale, the better). 

An original test based on a publicized British House debate showed remark- 
able unidimensionality of the original DQI components (Steenbergen et al. 2003). 
But in the context of parliamentary debates in various (institutional) contexts, the 
DQI attributes do not constitute a unidimensional phenomenon. A re-analysis of 
twenty-nine parliamentary debates at the level of speakers shows that there is a 
fairly strong correlation between sophisticated justifications and common good 
orientation (r = .61; p = .01); a medium correlation between respect towards 
demands/counterarguments and constructivity (r = .39; p = .04); and no or weak 
correlations between sophisticated justifications and respect towards demands and 
counterarguments (r = .08; p = .58), between sophisticated justifications and con- 
structivity (r = —.04; p = .83), between common good orientation and respect 
towards demands and counterarguments (r = .20; p = .29), as well as between 
common good orientation and constructivity (r = —.07; p = .73). These results are 
confirmed by a factor analysis, extracting two factors at the level of debates: one fac- 
tor combining sophisticated justifications and common good orientation and one 
factor combining respect towards demands/counterarguments and constructivity 
(see Table 6.1). 

The factor analysis indicates that deliberation in the real world is more com- 
plex than previously thought. One reason for this multidimensionality of deliber- 
ative quality is the public/non-public divide. While public debates accentuate the 
sophisticated justification/common good dimension, non-public debates accentuate 


Table 6.1 Factor Analysis of Deliberative Components in 29 Parliamentary Debates 
in Switzerland, United States, and Germany (see Steiner et al. 2004) 


Loading on Factor 1 Loading on Factor 2 

Sophisticated Justification 89 .01 

Common Good .90 .00 

Orientation 

Respect towards .11 .84 
demands/counterarguments 

Constructive Politics —.10 .84 

Eigen Value 1.62 1.40 


Note: Extraction Method: principal component analysis. Rotation Method: Varimax with Kaiser 
Normalization. 
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the respect and constructivity dimension. But if the elements of deliberative quality 
do not co-occur, we are left with a problem of interpreting what the different elements 
of deliberative quality actually mean. For instance, when actors justify their positions 
at length in a plenary debate but are simultaneously disrespectful, are we seeing 
a (moderate) contribution to deliberative quality (since they provided reasons), or 
are we observing ritualistic behaviour (Tanasoca and Sass 2019) or even (sophis- 
ticated) strategic behaviour? It is possible that a combination of ritualistic and 
strategic behaviour is at play here: since the norms of public debating in Western 
democracies generally value reasoned argument but not respect for the arguments 
of political opponents, strategic actors will use such mixed strategies (justification 
and disrespect). As such, the unidimensionality question touches upon a crucial 
validity problem: if there is no unidimensionality, then other action logics than true 
deliberation are present (Tanasoca and Sass 2019). 

Regarding ‘threshold’ values, Dryzek (2007, 244) has noted: ‘In applying the dis- 
course quality index, it is hard to say whether the deliberation in any of the cases 
analysed is actually good enough by any theoretical standards. The index is just a 
comparative measure: The threshold-level problem is intertwined with a level of anal- 
ysis problem. To date, the quality of deliberation is usually checked at the level of 
individual speeches (which is then aggregated to an average score for each speaker). 
But this is problematic: in order to achieve an overall maximum score, every speaker 
would not only have to justify their demands and arguments thoroughly in every 
single speech, but they would also have to be simultaneously oriented towards the 
common good and be respectful at all times. Even staunch advocates of delibera- 
tion might agree that this is impossible. In the context of Europolis, Gerber et al. 
(2018) have applied a holistic approach which analyses the overall deliberative per- 
formance of each speaker in an entire discussion. To do so, they have dichotomized 
the DQI indicators in accordance with pre-specified (and demanding) cut values. 
In concrete terms, they evaluated whether speakers offered a sophisticated rationale, 
referred to the common good, showed explicit respect towards other participants’ 
arguments as well as empathy to other groups, offered stories, and questioned what 
others have claimed—at least once in the discussion. In order to explore whether 
the pre-defined standards of classic and Habermasian-inspired deliberation repre- 
sent a latent variable of deliberative quality, Gerber et al. (2018) used (Bayesian) 
item response theory analysis(IRT)’. Deliberative ability is then constructed simi- 
larly to intelligence in educational science, namely with regard to how well citizens 
are able to achieve the various standards of deliberative quality (sophisticated jus- 
tification, explicit respect, etc.), while simultaneously assuming that some quality 
standards are more difficult to achieve than others. The empirical results are quite 
striking: in the context of Europolis, the deliberative ability of European citizens 
turns out to be a latent and unidimensional construct at the level of the speaker. This 
means that the various standards hang together,implying that a person who justifies 


1 A Bayesian approach using prior information is essential for model identification. 
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her positions at length also listens respectfully and is oriented towards the common 
good. Intriguingly, storytelling—while being the easiest standard—turned out to be 
a partial complement to justification rationality, that is, people who make sophisti- 
cated justifications also tell stories. As Gerber et al. (2018) concur: “This . . . provides 
a hint that the classic distinction between rational discourse and alternative forms of 
communication may be misleading, since high-skilled deliberators also use personal 
experiences to back up their positions and arguments: 

Another route to aggregate deliberative quality could follow the idea of ‘concept- 
driven’ lexical aggregation (Skaaning et al. 2015; Gerring et al. 2018). Lexical aggre- 
gation takes inspiration from John Rawls (1971) who orders the three principles of 
his theory of justice (liberty principle, fair equality of opportunity principle, and the 
difference principle) in order of ‘lexical’ priority: “That is, one should not consider B 
or C until A has been satisfied nor C until both A and B are satisfied’ (Gerring et al. 
2018). Notice first that lexical aggregation is generally similar to a (deterministic) 
Guttman scale and IRT (which adopts a probabilistic understanding of the relation- 
ship between a latent variable and an observed characteristic): a positive value for an 
item implies a positive value for all less difficult (or lower ranked) items (Gerring et al. 
2018;). Notice further that IRT employs a data-driven measurement model, requir- 
ing that the various items must co-vary. But when this is not the case, ‘researchers are 
at pains to solve the aggregation problem in a more deductive fashion’ (Gerring et al. 
2018). Consequently, Gerring et al. propose a non-Guttman-type ‘lexical scale’ which 
takes a ‘concept-driver’ or ‘deductive approach to aggregation’. Box 6.1 illustrates 
how the DQI could be turned into such a concept-driven lexical scale. 

Both IRT and lexical scaling understand deliberation as a ‘unitary’ phenomenon 
where the various quality standards complement one another in a cohesive whole. 
Notice, however, that the ‘unitary deliberator model has come under attack in 
the past decade. Several scholars have argued that deliberation should not be con- 
ceived of as a ‘single evaluative whole (Thompson 2008). According to Thompson 
(2008), aggregating deliberative quality into a single indicator may obscure the dis- 
tinct strengths and weaknesses of different deliberative standards. Similarly, Mutz 
(2008, 532) argues that it may be problematic to test deliberation as a compound 
phenomenon: 


[m]ost human behaviour is sufficiently complex that mere additive models are 
unlikely to account for it ... It would be not logical to ask how well a given 
deliberative encounter stacked up on all of the factors, create a combined score 
of deliberative goodness, and expect more beneficial outcomes associated with 
higher scores. 


? We thank Molly Scudder for helping us to clarify the distinction between different types of lexical 
scales. 
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Indeed, future research may be more interested in goal- and context-specific sensitive 
assessments of deliberative quality, requiring that we focus on more variegated con- 
figurations of the various standards of deliberative quality (Bachtiger and Parkinson 
2019), a point to which we turn in the next section. 


Box 6.1 ALexical Index of the DQI 


We make a first attempt to develop a lexical index for deliberative quality, based ona 
classical definition of deliberation focusing on extensive reason-giving and respect- 
ful listening. Focusing on the two crucial indicators of justification rationality and 
respectful listening, we construct an ordinal scale with five categories, running from 
not deliberative to high deliberative quality (Table 6.2). ‘Not deliberative’ means 
that participants provide no decent rationale for their positions. ‘Minimal delibera- 
tive’ quality means that a participant provides a decent rationale but does not refer 
to what other participants have said. ‘Medium deliberative’ quality implies that a 
participant both provides a decent rationale and makes a reference to other partic- 
ipants’ arguments. ‘High deliberative’ quality means that participants make both a 
sophisticated justification and a reference to other arguments. ‘Very high delibera- 
tive’ quality implies that there is a sophisticated justification as well as an explicitly 
respectful reference to other arguments. Finally, in the categories ‘very high’ delib- 
erative quality, both sophisticated reason-giving and respectful listening must be 
present. 


Table 6.2 Lexical Scale of Deliberative Quality 


0. Not Deliberative (in classic sense) ~J 

1. Minimal Deliberative Quality J =| 

2. Medium Deliberative Quality J | ~SJ 
3. High Deliberative Quality SJ l ~RI 
4. Very High Deliberative Quality SJ RI 


Note: 0-4 = ordinal scale; J = Justification, | = Interactivity, SJ = 
Sophisticated Justification, RI = Respectful Interactivity = conditions 
that are satisfied; ~ J, 1, SJ, RI = conditions that are not satisfied; 
empty cells = undefined. Relationships are deterministic. 


Using the Europolis database, we dichotomized all DQI indicators at the speaker level 
in accordance with these dimensions. The results are displayed in Table 6.3. 
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Table 6.3 Deliberative Lexical Scale 


Frequency Percent 


Not Deliberative 39 2167 
Minimal Deliberative Quality 9 5.0 
Medium Deliberative Quality 69 38.3 
High Deliberative Quality 13 72 
Very High Deliberative Quality 50 27.8 
N 180 100 


It is striking to see that a considerable number of participants (about 28 per cent) 
reached a very high quality of deliberation. This means that they were both capa- 
ble of formulating a sophisticated argument and referring to what other participants 
said in an explicitly respectful way. As a side note, correlating the IRT measure with 
our lexical index yields a score of 0.79. The high correlation score is not surprising 
since the IRT analysis reveals that deliberative quality formed a latent and unidimen- 
sional phenomenon in the Europolis discussions (Gerber et al. 2018). Consequently, 
a partly additive approach as inherent in the lexical approach will lead to similar 
results. But a lexical approach might be superior when deliberative quality is not a 
unidimensional phenomenon and the various indicators do not co-vary. 


The Future of Measuring Deliberative Quality: 
Challenges and Paths towards Automation 


Despite attempts to reconceptualize the DQI and align it better with current deliber- 
ative theory, two major challenges persist: an interpretivist as well as a functional and 
systemic challenge. This combines with a further methodological challenge, namely 
the demand—and possibility—to automatically code deliberative quality. 


Interpretivist Challenges 


Several scholars have questioned the validity of measuring the quality of deliber- 
ative processes on the basis of external and quantitative measurement. As King 
(2009, 9) argues, Habermasian discourse ethics ‘is interested in the intersubjective 
achievement of understanding a process of decision-making. Hence, a proper evalu- 
ation of deliberative quality must assess ‘how this is perceived by other participants— 
not merely the subjective speculations of outside observers. Bevir and Ansari (2012) 


André Bachtiger et al. 93 


have offered an even more vigorous attack on the quantification of deliberative qual- 
ity. They argue that if deliberation is meant to realize the goals of political legitimacy 
and emancipation, then scholars must treat deliberating actors as ‘intentional actors 
with the capacity for creative reasoning and agency’ (Bevir and Ansari 2012, 3). It is 
important to note that Bevir and Ansari do not fundamentally object to quantifying 
strategies, as long as coding efforts are based on careful interpretation by external 
coders. For instance, they acknowledge that the Discourse Quality Index ‘implicitly 
incorporate[s] interpretive approaches. 

Nonetheless, quantitative measures such as the DQI have largely obscured the 
meanings of deliberative acts. Interpretation is made by external coders and the 
focus of these instruments is usually on the variation of proportions of the different 
indicators of deliberative quality (whereby the size of the proportions is affected by 
different institutional incentives and norms). What is not considered is that the indi- 
cators of deliberative quality—such as justification rationality—may take on different 
meanings according to context. For instance, extended justifications in non-public 
settings may provide participants with good information and persuasive arguments. 
In the context of a plenary debate in a competitive and majoritarian system, how- 
ever, extended justifications may take on a very different meaning: they may serve 
to protect and promote the party’s ‘brand; and not be addressed to interlocutors at 
all (see Bachtiger and Parkinson 2019). Participants’ (and observers’) perceptions of 
the meanings of deliberative acts might vary as well. A respectful utterance may be 
perceived differently by different participants; or it may have multifaceted meanings 
which cannot be captured by a single code made by external coders. 

One possible way forward is to embrace more qualitative and interpretive 
approaches (Ercan et al. 2018). But we also think that it is misleading to put all of 
the eggs in the qualitative basket. While a purely quantitative approach will never 
be able to uncover the perceptions and meanings that participants attach to delib- 
erative and other communicative acts, a purely qualitative research strategy might 
be blind to deliberative dynamics as well as to complex transmissions in delibera- 
tive systems, which participants may not always see through (Bachtiger 2018). As 
we shall discuss below, this requires computer-assisted tools and big data analyses to 
understand transmissions and dynamics happening at a supra-individual level. 


Functional and Systemic Challenges 


Recent developments in deliberative theory apply a problem-based approach to 
conceptualizing and measuring deliberation (Bachtiger and Parkinson 2019). A 
problem-based approach understands deliberative quality as contingent, that is, 
dependent on different contexts and goals, as well as distributed across space and 
time. This means, for instance, that we cannot apply the same deliberative qual- 
ity indicators for plenary debates in parliaments and deliberative mini-publics: in 
the former context, what counts is the quality of argument to realize a parliament's 
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accountability function, whereas in the latter context, respectful interactions may be 
crucial for creating a civic spirit amongst citizens. From sucha functional perspective, 
the time ‘for a grand, unified index of deliberative quality is over’ (Bachtiger and 
Parkinson 2019, 138). When deliberative goods, such as reason-giving or listening, 
are contingent on goals and contexts of deliberation and can be distributed across 
space and time, then ‘measuring’ deliberative quality cannot be done on the basis 
of an index which aggregates fixed deliberative standards found in a communica- 
tive sequence of a single forum. The right number of indicators depends on goals 
and contexts, and thus becomes flexible and changeable. This will require novel and 
more theoretically grounded ways of aggregating deliberative quality. 


Towards Automation 


For a long time, studying the processes of deliberation empirically was a highly 
demanding and time-consuming endeavour, requiring in-depth content analysis and 
extensive reliability testing. Computer-assisted textual analysis can help to speed up 
data collection and supersedes reliability testing. The automation of the DQI was 
taken recently with DelibAnalysis, which used machine learning to derive scores 
for online deliberations with politicians (Fournier-Tombs and Di Marzo Serugendo 
2019), as well as an updated version for parliamentary deliberations in the three 
Canadian Territories (MacKenzie and Fournier-Tombs 2019). The machine-learning 
approach employed in the original DelibAnalysis used a so-called classification algo- 
rithm to categorize DQI scores into either low, medium, or high scores: in order to 
obtain the scores, a small subset of data (usually < 1%) was manually coded by the 
researchers, in order to provide parameters for the quality of discourse in a given 
context. From that, the algorithm was trained, predicting scores for the rest of the 
comments in the dataset. In the updated version of this methodology, the researchers 
created a model for each indicator in the DQI. Unlike the human coder who uses a 
coding manual to determine the DQI scores, the model uses quantitative features, 
such as the presence of phrases or parts-of-speech and character counts. As a general 
rule, the models generated for these studies had an 80 per cent or higher accuracy, 
with some models reaching upwards of 95 per cent accuracy. Although this approach 
requires some manual intervention at the onset, it benefits from flexibility, allowing 
researchers to measure quality in different contexts, languages, and platforms, with- 
out relying on a rules-based programming method. In sum, automation will be an 
indispensable part of analysing deliberative quality in the future, especially when it 
comes to assessing systemic deliberation. 


Conclusion 


The DQI and other measurements of deliberative quality have contributed to mak- 
ing a normative concept amenable to empirical research. Yet, as we have seen in this 
chapter, this translation is anything but simple, and requires constant updating of 
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empirical strategies in order to meet demanding—as well as changing—normative 
standards and approaches. Indeed, future research on deliberative quality calls for 
more sophisticated empirical strategies: strategies that take into account elements of 
the goal- and context-dependency of deliberative forms and that capture the distri- 
bution of deliberative acts across space and time. Yet, somewhat paradoxically, the 
fact that we are currently experiencing a crisis of public communication (Dryzek 
et al. 2019)—including the rise in simplifying and disrespectful populist language— 
has turned both high-quality argumentation and respectful listening into a new focal 
point for assessing political discourse. Its manifold blind spots notwithstanding, the 
original DQI provides an excellent diagnostic tool with which to analyse high-quality 
argumentation and respectful listening (see Marien et al. 2019). 
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7 
Deliberative Reason Index 


Simon Niemeyer and Francesco Veri 


The Deliberative Reason Index (DRI) offers a substantially different method for 
evaluating the effects of deliberation compared to its counterparts. Its outstanding 
feature concerns an approach to analysis of the content of reasoning as a group- 
level reasoning, resulting in a property of deliberative reason that is greater than the 
sum of reasoning by individuals. The development of the method reflects a grow- 
ing recognition that group reasoning is an metaconsensus intrinsic feature of human 
cognition—sometimes for worse (Sloman and Fernbach 2017); but also for better, 
if a group context is sufficiently deliberative (Mercier and Sperber 2011; Chambers 
2018). The method also provides a measure of how well the group and the individuals 
that comprise it have contributed to improvements in reasoning. 

Although DRI is a relatively recent development, its history dates as far back as 
2000, when its empirical precursor, intersubjective consistency, was first observed as 
part of the Bloomfield Track study (reported below). Over the past two decades it has 
been subject to verification and refinement, including the development of a method 
for aggregating and indexing. 

Deliberative reason, as measured by DRI, involves mutually constructing an 
understanding (or shared ‘logic’) of what is at stake for an issue and establishing 
the contours of ‘reasonableness’ Individual positions may diverge, but on terms that 
are mutually agreed. Those terms include higher-level agreement concerning rel- 
evant interests, values, and beliefs (metaconsensus; Niemeyer and Dryzek 2007), 
as well as coherence in the form of a mutually shared ‘representation’ of the issue 
(Mercier and Sperber 2011) guiding how considerations inform preferences. Empir- 
ically, the effect of these conditions produces intersubjective consistency, measured 
as proportionality between reasonable disagreement (or agreement) among pairs of 
individuals regarding considerations on one level and preferences on the other. DRI 
is the group-level aggregation of this effect. 

Measuring DRI first requires the development and deployment ofa multi-level sur- 
vey instrument capable of capturing intersubjective consistency—soliciting opinions 
regarding issue considerations on one level, and preferences regarding possible poli- 
cies or actions on the other. The degree of consistency between these levels amongst 
individuals in the group is then analysed and aggregated to produce DRI. 

DRI can be used to analyse the extent to which group reasoning has improved dur- 
ing deliberation and the conditions that best facilitate it (e.g. Niemeyer et al. 2021). It 
also lends itself to detailed interpretive analysis of how individuals variably contribute 
Simon Niemeyer and Francesco Veri, Deliberative Reason Index. In: Research Methods in Deliberative Democracy. 
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to group reason and how these dynamics connect to wider political discourse, 
as well as understanding the mechanisms that improve reason in wider delibera- 
tive systems. Empirical applications include multifactor analysis across deliberative 
cases and comparative analysis of deliberative reason between different cohorts, as 
demonstrated below. 

This chapter elaborates briefly on the theoretical foundations of the method, before 
outlining the process of implementation and interpreting results. It then demon- 
strates how DRI captures the impacts on group reason, before outlining its limitations 
and providing concluding remarks. 


Deliberative Reason and DRI 


The method for obtaining DRI builds on an understanding of deliberative reason 
as inherently a group process involving mutual communication regarding relevant 
considerations about matters of common concern to arrive at conclusions regard- 
ing what should be done (Bachtiger et al. 2018). Reasoning under ideal deliberative 
conditions of openness, reciprocity, and sincerity involves a process of justifica- 
tion, bringing arguments, interests, and concerns to light, and holding individuals in 
the group accountable to the conclusions that they draw. The overall effect induces 
‘enlarged thinking’ (Arendt 1961) that expands the domain of moral consideration 
and incorporation of shared knowledge into reason. 

The process of deliberative reason, thus described, also results in improvement in 
integrative reasoning, reflecting a shared appreciation of the world that deliberators 
cohabit, the meanings that they imbue from it, and the various interests impacted by 
the choices they make—without necessarily agreeing (for good reasons) on should 
be done. Their efforts to achieve integration take account of those considerations that 
are raised via deliberation. 

Integrative reason(ing) here involves a more demanding version of the psycho- 
logical construct of integrative complexity, which measures reasoning in terms of 
differentiation and integration of perspectives (Tetlock 1986). Higher integration 
results in more sophisticated forms of reasoning that synthesize and accommodate 
alternative views (Wyss et al. 2015), creating a more inclusive politics (Wright 2019). 
In addition to sophistication, deliberative reason benchmarks the level of integra- 
tion into reasoning across’ the range of considerations that deliberators mutually 
agree to be relevant—even if they do not agree on their veracity or importance. 
The imperative to account for these considerations acquires a moral force via the 
deliberative ideals of respect and reciprocity (Gutmann and Thompson 1996) in not 
only recognizing those impacted by a decision, but accommodating their interests 


* This agreement on relevant considerations constitutes a form of meta-consensus (Dryzek and 
Niemeyer 2006). 
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into reasoning alongside all relevant considerations identified by the group without 
exception (Misak 2004). 

Deliberative reason also involves the formation of coherence (Davidson 2001), such 
that group members understand each other’s justifications. This shared reasoning 
holds individuals accountable to a mutual group understanding of the world and the 
internal values that inform their judgements. Any revealed dissonance leads either 
to the group accommodating that individual’s perspective or adjusting their under- 
standing according to a common standard of reasonableness, which regulates what 
constitutes a good argument. 

The net effect of this deliberative process is to widen the field of view of individ- 
uals, who take seriously their mutually identified set of relevant considerations. It 
also produces a shared justificatory basis whereby judgements regarding claims of 
fact and relative priorities of values inform decisions regarding what should be done 
(preferences)—resulting in a mutually shared ‘representation of the issue regarding 
how premises connect to conclusions (Mercier and Sperber 2011). 

A representation may be partly formed before a given deliberative encounter 
and subject to revision or evolve as a product of deliberation. Deliberative reason 
accommodates contestation across overlapping issue representations—so long as 
they share a working meta-consensus (Dryzek and Niemeyer 2006) on relevant issue 
considerations—accommodating and synthesizing different forms of reason struc- 
tures (e.g. across different cultural settings) without necessarily requiring agreement 
on outcomes. 

Take, for example, the challenge of constructing an effective policy response to a 
pandemic, such as COVID- 19. Relevant considerations informing what to do include 
epidemiological knowledge concerning the transmissibility, mortality, and morbid- 
ity of the virus and relevant risk factors. Other considerations might concern the 
economic impacts associated with the use of available policy levers, such as lock- 
downs. Certain prior representations of the issue may treat these considerations as 
legitimate trade-offs, leading some to support a policy of herd immunity, given the 
uncertain prospect of an effective vaccine at the time. Other considerations, such as 
restricted freedom of movement contributing to social and psychological impact may 
also factor in reasoning. Representations incorporate contested facts, such as those 
concerning the effectiveness of certain policies. Deliberation improves knowledge 
and its incorporation into a revised representational framework—improving epis- 
temic performance (Landemore 2017), settling contested claims, dispelling misinfor- 
mation (Niemeyer 2011), or questioning established wisdom in light of arguments. 
For example, arguments in favour of herd immunity may lose validity in the face 
of growing evidence that economic performance depends on confidence in public 
safety, or the emergence of effective vaccines, with either development reconfiguring 
representational understanding. Deliberation might also serve to remind us of moral 
obligation to the aged and immunocompromised, producing representational adjust- 
ments that reframe understanding of the term ‘dies with COVID to imply greater 
collective responsibility for those affected. 
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Whatever its form a shared representational framework constructed during 
deliberation should render all aspects of an issue mutually intelligible, along with 
acceptance of relevant values, and implications of actions where facts are settled. 
Where they are not, a deliberatively shared framework accommodates diversity, as 
well as creating conditions for resolving contested elements (Landemore 2017). 

Deliberative reason in the longer term involves ongoing updating of represen- 
tational frameworks, which gain sophistication as more facts become settled and 
enlarged thinking accommodates a growing set of relevant considerations. In the 
shorter term it pushes individuals towards group coherence, on terms they them- 
selves set, synthesizing disparate elements of representations in the formation of 
positions as they come to reason together, and in doing so facilitating greater levels 
of trust (Mercier and Sperber 2011). 

The effect of deliberative reason described above is the formation of a mutually 
constructed representation that accommodates a ‘reasonable pluralism. The result is 
a shared rationality—literally, in the sense that it embodies a proportionality that 
can be observed at the group level. Well-formed positions are deliberative to the 
extent that they are consistent with that structure, which is capable of integrat- 
ing all relevant arguments, even if only to establish grounds for disagreement that 
can be mutually endorsed as reasonable. This reasoning may involve some mea- 
sure of tacit knowledge (Benson 2019) accessed through intuitive reason (Habermas 
2017). But it nonetheless produces regularities of reason within the group such that 
a given level of disagreement on opinions regarding considerations is regulated via 
a shared representational framework, resulting in proportionality between prefer- 
ence/consideration agreement level, or intersubjective consistency (Niemeyer and 
Dryzek 2007).” It is this effect that forms the basis of DRI. 


Measuring Deliberative Reason 


The measurement of DRI involves the development and use of a survey designed to 
capture opinions regarding the relevant issue considerations on one level, and pref- 
erences for actions or policies on another. The items used to survey considerations 
usually take the form of statements sampled from the relevant public debate—where 
it is important that the instrument reflects the working meta-consensus, or range of 
considerations deemed relevant to the issue.’ 

Opinions regarding considerations can be surveyed using a Likert response along 
an agree-disagree scale, although higher quality data is obtained using quasi- 
ranking. Most of the studies reported below have used both methods when surveying 


? For a more detailed explanation of this effect, see Niemeyer et al. (2021). 

° A requirement reflecting the operation of meta-consensus (Dryzek and Niemeyer 2006). Although 
DRI is an entirely separate method to Q methodology, it shares an approach with best practice for collect- 
ing and developing survey items (Niemeyer 2020), where Q methodology uses the language of ‘concourse’ 
instead of meta-consensus (Stephenson 1986). 
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responses.* The approach involves first obtaining a Likert response, which can be 
optionally be used as the basis for beginning the process of sorting statements into 
a series of (usually up to eleven) categories (columns) from most disagree to most 
agree.” Surveying preferences involves the ranking of a set of options (usually between 
five and ten) in the form of implementable actions or policies.® 

Once opinions and preferences have been surveyed, DRI is obtained via four steps. 
The first step involves correlating, in turn, responses to the survey of considerations, 
and a preference survey, for pairs of individuals (using Spearman correlation; p).” In 
the second step, the level of consistency for all pairs of individuals is calculated. The 
third step calculates DRI for each individual. Finally, in the fourth step, the result is 
aggregated to produce DRI for the group. 

The first two steps of the process are explained using Figure 7.1. The figure plots 
intersubjective consistency for four sample individuals (A,B,C,D) drawn from the 
Bloomfield Track case study—concerning a controversial road constructed in a 
World Heritage Listed rainforest (Niemeyer 2004).° In step 1, the intersubjective cor- 
relations for both considerations ( pr) and preferences ( pp ref ) are calculated for all 
sets of pairs (J = [AB, AC, AD, BC, BD, CD])—where, for = 3 individuals in the 
example, the total number of pairs (nz) (nz = mn) = 6). The figure plots these 


cons 


pairs, with the corresponding values for the pair AB (p°)* and p H re ) illustrated using 
dotted lines. 
Step 2 calculates the distance of all pairs from the 1:1 (intersubjective consistency) 


line, shown in Figure 7.1. The modal orthogonal distance (d) is obtained for a given 
pair (j) using dj = | pf ge p| v2). In the figure, the distances for all our example 
pairs (da, v, da, c, da, d, Abc, de, a, de, q) are indicated using dashed lines. 

Step 3 then involves finding individual DRI (DRI”®) for each member of the group 
(i), beginning with finding the average distance and then transforming the result into 
a —1 to 1 scale. 


The average modal distance (D;) is found using the formula: 


_ ft ao 
D; = a >, dj for all pairs that include individual i; 
m 


(Jj... m; where m; =n — 1) 


* For a detailed list of these studies, see Niemeyer et al. (2021). 

° Surveying can either involve a specifically designed online survey facility or be performed manually 
using cards that are sorted into a distribution (see Niemeyer 2020). 

Ś For some cases, the outcome options are already established as part of the mini-public design; in 
others, those options are developed as part of the deliberative process. There is evidence that designing a 
deliberative process to develop options has a more positive effect on deliberative reason than voting on 
pre-established options, which potentially has a negative impact. See Niemeyer et al. (2021). 

7 We employ Spearman p because it is particularly appropriate to correlate non-linear relationships 
and ordinal data, such as the Likert scale’s ranks (Lehman et al. 2013, 127). Java-based software is available 
(DeliberateQ). It can be downloaded at https://github.com/DeliberativeAnalysis/DeliberateQ/releases. 

® The points A,B,C,D correspond to participants ‘ASW; ‘KEI; JAN’, and ‘TAM’ from the study. The data 
and the survey instrument are available at https://participedia.net/case/38. The data are also built into the 
DeliberateQ software, which can be used to replicate the results (see note 10). 
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Intersubjective Agreement — Preferences 
(Spearman p) 


Intersubjective Agreement — Considerations 
(Spearman p) 


Figure 7.1 Intersubjective Consistency: Illustrative Plot 


Using individual A from Figure 7.1 as an example, the average modal distance (Dz) 
is obtained for all pairs that A is part of (Ja = [da », da, c da, al). 

Individual DRI (DRI) is then calculated by transforming D; into a —1 to 1 scale. 
This is done using the theoretical maximum average distance for a given group—or 
Lambda (A = Dmax), where À is graphically indicated in Figure 7.1.° The resulting 
formula for individual i is 


DRI” —2D; +À 
ro A 


° Because of the interrelationships between pairs, A is constrained to a value less than maximum geo- 
metric distance (\/2/2 or 0.71) for any case where n>2. Even if the distance for some points is greater, the 
average quickly asymptotes on A as n—+00. To illustrate, if we take individuals X and Y who are in perfectly 
inconsistent agreement (e.g. plotting at the 1,0 in Figure 7.1), such that dx, = 0.71, and we then add an 
additional individual (Z) who is in perfectly inconsistent in disagreement with X (d,,z = 0.7) they would, 
by definition, be in perfectly consistent disagreement with Y (d,,, = 0), such that the maximum average 


distance (A) declines from 0.71 to 0.47, and so on until A + /2/2. 
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Finally, DRI is calculated for the group by averaging DRI” using 


ls d =i 
DRI = F XO DRI” for alli (i... n) 


i=1 


Note that it is possible to use a shorter method for calculating DRI that does not 
require calculating DRI’, by obtaining and averaging D; for all points in step 3. 
However, as discussed below, using DRI”? permits greater interpretive and analytical 
possibilities. 


Interpreting DRI 


The previous discussion illustrated the overall relationship between DRI and the aver- 
age distance of its constitutive points (i.e. of individual pairs). This relationship is 
geometrically illustrated for A,B,C,D in Figure 7.1, where the overall average distance 
is shown as the darker, inner band running parallel to the intersubjective consistency 
line. The boundary of this band is slightly greater than half the maximum average dis- 
tance (D = \/2) which equates to a DRI slightly less than 0.’° A similar result occurs 
when all twelve participants from the Bloomfield Track study are added to the pre- 
deliberative plot, as shown in Figure 7.2 (DRI pos = —0.2)—where the average distance 
is also indicated by a dark band, and its intersection with the y-axis projected onto 
the DRI graph located between the pre- and post-deliberative plots. 

A DRI of 0(D = \/2) suggests a complete absence of consistency across the group 
as a whole—which might occur, for example, where complete non-attitudes apply to 
all considerations across all individuals, or extreme partisan framing induces very 
little integration beyond narrow or irrelevant considerations. 

A DRI approaching —1 (d = A) reflects cases of perfect inconsistency in reason, 
for example, where wilful partisanship produces reason using a representation that 
inverts the conclusions that would be drawn from the same opinions—a very unlikely 
outcome. By contrast, a DRI of 1 represents perfectly consistent reason, where all 
points fall precisely on the intersubjective consistency line in Figure 7.1. In prac- 
tice, most observations of DRI fall above 0 (see Figure 7.3). Care should be taken 
when interpreting overall group DRI, particularly for values that fall near DRI = 0. 
Such cases may involve variability in levels of reason beneath the headline figure that 
produces a similar overall result—for example, where a small subgroup with very 
low DRI”? disproportionally impacts the group result (see the discussion of cohort 
analysis below). 

Take the example from Figure 7.1, where individuals A,B, and C collectively exhibit 
relatively high levels of agreement on both considerations and preferences—their 


1° An average distance of À represents DRI = —1. DRI = 1 where all points fall on the intersubjective 
consistency line (D = 0). 
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points (AB, AC, BC) plotting towards the upper right-hand quadrant of the figure and 
close to the intersubjective consistency line. Their subgroup DRI is relatively high, 
and so too are their consensus levels, indicating a reasonable agreement in terms of 
preferred outcome. 

Importantly, consensus does not connote reason. The pair AB have the strongest 
agreement on preferences, but are slightly less consistent compared to the pairs AC 
and BC. Nevertheless, the interaction between consensus and reason can be used 
when interpreting the results. Take, for example, participant D who has a much lower 
DRI” than the rest of the group. While D shares relatively high consideration agree- 
ment, this does not translate into the comparably high levels of disagreement on 
preferences. This inconsistency effect, combined with the pre-deliberative consen- 
sus regarding considerations, and the fact that D has a disproportionately disruptive 
effect on DRI, all contribute to the interpretation of reason for the wider Bloomfield 
Track Citizen's Jury case, which is conducted below. 


Interpreting Deliberative Reason: The Bloomfield 
Track Example 


The pre- and post-deliberative plots in Figure 7.2 add the remaining pair combi- 
nations of the twelve individuals for the Bloomfield Track case to those illustrated 
in Figure 7.1.” The pre-deliberative plot on the left-hand side of the figure shows a 
similar distribution of all pairs along the right-hand side of the y-axis to Figure 7.1. 
This is typical of most of the pre-deliberative studies analysed and reported below 
in Figure 7.3, reflecting a similar pattern to that achieved using randomly generated 
values (Monte Carlo simulation), but with a distribution to the right of the y-axis, 
reflecting a non-random level of agreement on considerations. 

In the case of the Bloomfield Track, the explanation of this distribution is the dom- 
ination of the issue by misinformation which, before deliberation, induced partisan 
reason and polarization (Niemeyer 2011). Before deliberation, there was a strong 
pre-existing agreement on a range of considerations—such as the ecological preser- 
vation of significant rainforest and reef environments surrounding the road, which 
did not translate into comparable agreement on preferences—across options, rang- 
ing from upgrading the crudely built track to a conventional road, to closing it and 
rehabilitating back to rainforest. 

The explanation for this inconsistency involves the effect of partisan framing, 
which dominated the politics of the issue, creating salience effects that served to frac- 
ture underlying consensus. It did so by harnessing narrow but symbolically important 


1 Although agreement on outcomes should be interpreted with care, ideally drawing on a detailed 
understanding of the operating preference meta-consensus (see Niemeyer 2020). 

12 Plotting all combinations of pairs for all twelve participants n, we end up with n(n — 1) /2 = 66 
points. 
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Figure 7.2 IC Plots: The Bloomfield Track (Far North Queensland Citizens’ Jury) 


arguments in order to mobilize support for the outcomes proposed by partisan actors 
(Niemeyer 2004). The outcome was poor deliberative reason, reduced integration, 
and polarized preferences. 

The impact of deliberation in remediating these partisan frames is illustrated in 
the post-deliberative plot for the study on the right-hand side of Figure 7.2 (where 
DRI improves from —0.02 to 0.49). Notably, transformation involved relatively lit- 
tle opinion change, except for those considerations subjected to misinformation 
before deliberation. Preference transformation amplified these relatively small opin- 
ion changes, with reduced impact of misinformation attenuating salience effects, 
leading to greater integration of all considerations into reason. The effect dramati- 
cally changed the overall outcome, from favouring upgrading to a strong majority in 
favour of closing the road altogether (Niemeyer 2004). 

The nature of this effect demonstrates the limitations of using opinion change alone 
as a proxy for deliberative reason. Opinion change on considerations can be observed 
in Figure 7.2 via the relatively small level of horizontal movement of pairs compared 
to very strong vertical movement (preference transformation) using the individuals 
A,B,C,D from Figure 7.1.'* For this group, much of the change is attributable to D, 
although the effect was more evenly distributed across the case as a whole. 

The result is a post-deliberative preference consensus reflecting a pre-existing con- 
sideration consensus, and a much higher DRI, driven by improvement in deliberative 
reason. Importantly, this effect was not only due to improvement in reason by the 
outliers. The wider group also became attuned to those concerns that had been 
most amenable to symbolic manipulation (Niemeyer 2004). Rather than dismiss- 
ing them, they accommodated and integrated them into a shared representation, 


13 The animation of this transformation can be observed at https://youtu.be/8UQ4hPvzr2Y. 
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thus discursively creating the conditions of trust necessary to combat the impact of 
misinformation. 

Preference consensus was incomplete following deliberation. There remained 
a small group after deliberation that required more evidence before endorsing 
road closure. However, a well-developed common representation of the issue post- 
deliberation meant that differences could be understood and accepted by all as 
reasonable (Niemeyer 2004). 


DRI in Deliberative Democracy Research 


The Bloomfield Track example above demonstrates the use of DRI as a tool for 
analysis of deliberative reason. The method uses similar survey-based inputs to 
those traditionally used to measure deliberative opinion transformation (e.g. Lindell 
et al. 2017), where DRI avoids the problem where simple opinion transformation 
may occur for non-deliberative reasons, such as increased polarization. DRI can be 
applied to a broad range of research design settings, different scales, and for drawing 
inferences about reason regarding individuals (in relation to the group), groups, and 
subgroups. The discussion below outlines some of the possibilities. 


Inferential Applications of DRI 


The Bloomfield Track case represents a straightforward example of the application of 
DRI, analysing changes to deliberative reason during mini-public deliberation. DRI 
is measured at multiple stages during the deliberative process. Ideally, the confidence 
in findings is improved when analysis is paired with a control group performing the 
same pre- and post-surveys without deliberating. This is the case for the Uppsala 
Speaks and AusCJ studies in Figure 7.3, where arrows indicate the direction and mag- 
nitude of change (pre- to post-deliberation). DRI does not change significantly for 
either of these control cases, compared to the significant changes to their deliberative 
counterparts (as indicated by asterisks). 


Comparisons across Cases 


The use of DRI for testing improvement in group reason is also demonstrated in 
Figure 7.3. The cases in the figure are divided into two groups based on their 
approach to ‘group building: The vast majority of nineteen deliberative cases expe- 
rience improvement, but the effect is variable. Much of this variability is accounted 
for by differences in group building, where the group-building treatment produces 
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Figure 7.3 Summary DRI Results 
Note: One-tailed Wilcoxon Test: p<0.05*, p<0.025**, p<0.01***. 


significantly higher group reason than the standard group. This effect was specifi- 
cally tested using an experimental design for the Uppsala Speaks, where DRI was 
capable of discriminating between design effects, and the control, unlike opinion 
transformation.” 


DRI and Analysis of Causation 


DRI can also be used as a variable (dependent or independent) to establish causal 
relationships in more complex modelling. Structural equation modelling can be 
used to estimate the mediating effects between factors that characterize participants, 
the deliberative process itself, and DRI. Using multi-level regression, causal influ- 
ences on DRI can be explored using characteristics at the individual level (such as 
demographic, socio-economic or cultural characteristics), as well as case-level fea- 
tures (such as process design, issue features, or even contextual variables such as 
country or political system) (Niemeyer et al. 2021). 

A potential limitation with DRI includes commensurability between studies. 
This does not apply to the Uppsala Speaks study, which involved an experimen- 
tal treatment with cases that share the same issue and survey instrument. However, 
other comparative analysis may involve different instruments, potentially impacting 
robustness. One way to overcome this limitation is to structure multiple regression 
analyses using a model that nests individuals within the cases, thus holding the trans- 
formation effects constant. This analysis was performed in Niemeyer et al. (2021), 
which used changes to DRI'"“” as a dependent variable to analyse improvements 


14 Significance of DRI change was tested using non-parametric version of the t-test (Wilcoxon test) and 
one-way ANOVA (Mann-Whitney and Kruskal Wallis). See Niemeyer et al. (2021). 
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to deliberative reason for nineteen cases of mini-public deliberation reported in 
Figure 7.3 using individual- and case-level variables, including group building.’° 


Cohort DRI Analysis 


DRI can also be used comparatively to explore deliberative reason across different 
cohorts of individuals as separate groups, connected by the same issue and survey 
instrument. Cohorts (or enclaves) may be delineated within a given target popula- 
tion, or they may be drawn from different populations separated by some boundary 
such as a national border. These cohorts may each separately exhibit high levels 
of deliberative reason, but not when combined as a group (e.g. across cultural dif- 
ferences), ‘talking past’ each other in the absence of an intersubjectively shared 
representation of the issue. 

The approach to testing cohorts can be illustrated by the most right-hand case 
shown in Figure 7.3 which was part of a wider study on Climate Change and delib- 
eration (Hobson and Niemeyer 2011). The study tests differences in DRI between 
two cohorts, the first involving self-identified climate sceptics (n = 43) which were 
compared to their non-sceptic counterparts (n = 59) who were all surveyed prior to 
deliberation regarding their positions (Hobson and Niemeyer 2013). 

The potential sources of differences between sceptics and non-sceptics can be 
understood in terms of deliberative reason via three possibilities: (1) reasonable 
disagreement where sceptics and non-sceptics metaconsensually share a represen- 
tation of the matter—disagreeing on conclusions but in a manner consistent with 
deliberative reason; (2) enclave deliberation, where the groups develop different 
representations such that sceptics and non-sceptics share an internal reason struc- 
ture within their own discourses, but not across the two cohorts; and (3) variable 
deliberative reason, at least one group exhibits a lower standard of group reason. 

Analysis of DRI across the sceptic and non-sceptic cohorts supports the third of the 
three hypotheses: that there is variable performance in deliberative reason, with the 
climate sceptics cohort (DRI = 0.39) performing significantly more poorly than their 
non-sceptic counterparts (DRI = 0.59) (p<0.001). The result, illustrated in Figure 7.4, 
is consistent with a growing body of evidence that the reason of deep sceptics (see 
Hobson and Niemeyer 2013) is more likely to involve deliberative pathologies such as 
motivated reason (Whitmarsh 2011). Overall, deeper varieties of sceptic are less likely 
to engage in deliberative reason, implying a lower DRI, which was reflected in their 
behaviour when participating in the deliberative process (Hobson and Niemeyer 
2013). 


13 The multi-level modelling and the multi-level moderator analysis support the group-building 
hypothesis discussed above, as well as identifying features such as use of voting for transmitting out- 
comes versus reasoned reports on group reason or issue complexity—though these effects are themselves 
moderated by use of group building. 
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Figure 7.4 DRI Analysis of Climate Sceptics versus Non-Climate Sceptics 


Methodological Limitations and Future Developments 


Despite DRI’s analytical power, there are limitations associated with the method. 
As discussed earlier, direct comparability of DRI scores is impaired between cases 
that do not share the same survey instrument—although this is surmountable using 
specific methods (see above). Poor instrument design may also skew results. For 
example, a high pre-deliberative score may result from a survey instrument design 
involving high-level or generalized statements for which it is relatively easy to find 
common basis of reason, but which do not necessarily reflect the substance of the 
issue. Alternatively, an artificially low DRI may result from a poorly designed instru- 
ment that includes considerations or policy options that are irrelevant, or omits 
others that are pivotal, such that it does not adequately reflect considerations or 
options that are relevant to the group (i.e. failure to reflect meta-consensus). 

These limitations reinforce the need to develop a survey instrument that is 
appropriate for a given issue. The survey instrument must not only capture a pre- 
vailing meta-consensus, it also needs to do so at the appropriate level of resolution. 
Careful attention to survey development is particularly important where the issue at 
hand is complex or where there is relatively little existing public debate from which 
to sample statements. 

Depending on the study and complexity of the topic, implementing DRI can be rel- 
atively demanding. Development of the survey instrument ideally involves a series of 
pilot tests to assess how well it covers the relevant considerations and action options. 
Analysis and interpretation of results also requires care. As discussed above, the DRI 
method works best when combined with strong qualitative and interpretive skills. 

However, the DRI method produces strong results, and (we argue) provides a more 
theoretically defensible approach to measuring deliberative outcomes than standard 
survey methods (Dryzek 2005). Further testing and development will help to simplify 
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the task of survey design, and potentially reduce the number of items required to cap- 
ture deliberative reason with confidence. Further reliability testing will also render 
the method suitable for wider application, using shorter survey formats. 


Conclusion 


When properly developed and applied, DRI represents a powerful approach for 
opening the black box of deliberative reason(ing), with the potential to inform 
our understanding of the processes involved and the implications for deliberative 
theory and practice. Further development will lead to an updating of the method— 
providing greater certainty regarding its application, addressing sensitivity to the 
survey instrument, and permitting easier application in wider settings. 

Although DRI involves abstraction of group reason to produce a single index, the 
method works best when the researcher seeks to understand what is at stake from the 
perspective of the survey respondent, particularly when developing the survey instru- 
ment. The results can also be used interpretively to understand variations occurring 
within the group, and the various factors driving them to arrive at their conclusions at 
different deliberative stages. In this sense DRI is an authentically deliberative research 
instrument, where analysis ideally extends to understanding the dynamics that have 
produced a given DRI result. 
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8 
Listening Quality Index 


Mary F. Scudder 


Listening is at the heart of meaningfully democratic deliberation (Bickford 1996; 
Dobson 2014; Scudder 2020). Yet empirical researchers of deliberative democracy 
have struggled to provide an adequate means of studying listening in practice (Eve- 
land et al. 2020). The Listening Quality Index (LQI) presented in this chapter seeks to 
fill this gap by providing a theoretically informed, normatively relevant, and empiri- 
cally feasible measure of listening to be used in the empirical study of small-scale or 
‘micro’ deliberation.” 

One key challenge to assessing listening in micro deliberation is observation. Ulti- 
mately, listening is an internal act and can only be measured indirectly by looking for 
observable behaviours that we might expect to correspond with listening. But even 
‘silence, which is readily observable and perhaps a necessary condition for listening, 
is just as likely to signify withdrawal from discourse as it is meaningful engagement 
(Bickford 1996, 154). Given these observational challenges, empirical deliberation 
research has relied primarily on ‘responsiveness’ as a proxy for listening (Steenbergen 
et al. 2003; Steiner 2012; Button and Garrett 2016). But there are limitations to this 
approach. By equating listening with responsiveness, existing accounts miss impor- 
tant variations in listening and listening quality. On the other hand, measures that 
equate listening with its antecedent conditions, or simply the opportunity to hear, 
run the risk of overestimating the amount of listening taking place. 

The LQI addresses these challenges of measuring listening by combining various 
listening attributes into a lexical scale. Lexical scales use a cumulative logic, where 
each attribute included in the measure is seen as a necessary condition for advanc- 
ing up the ordinal scale. Drawing on insights from normative political theory, the 
LQI identifies relevant listening indicators, for example, recall or responsiveness, and 
then arrays these indicators along a continuum where each level represents a dis- 
tinct step towards democratic listening. The LQI improves upon existing measures by 
allowing researchers to capture more variation in listening while also making quality 


* Listening is just as important in macro deliberation, or the communication that occurs in broad, infor- 
mal public spheres (Bachtiger and Parkinson 2019; Scudder 2020). But assessing listening quality in that 
context requires a different set of strategies (Scudder 2020, 140-143). The LQ] is a tool designed specifically 
for micro deliberation including citizens’ assemblies, mini-publics, parliamentary debates, or experiments, 
either online or face-to-face. A portion of this chapter draws on Mary F. Scudder, ‘Measuring Democratic 
Listening: A Listening Quality Index’ which was published in Political Research Quarterly 75, no. 1 (2022). 
I am grateful for permission to reuse that material here. 


Mary F. Scudder, Listening Quality Index. In: Research Methods in Deliberative Democracy. Edited by Selen A. Ercan, 
Hans Asenbaum, Nicole Curato, and Ricardo F. Mendonca, Oxford University Press. © Oxford University Press (2022). 
DOI: 10.1093/0s0/9780192848925.003.0008 
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assessments of listening behaviours. For example, the LQI tells us how we should 
compare a situation where a listener responded to a speaker, but interrupted, and a 
situation where a listener did not respond, but allowed the speaker to finish speak- 
ing. In this chapter, I first outline the existing measures of listening before presenting 
the LQI and explaining how it can be used to measure the quality of listening in 
deliberative processes as well as in other communicative interactions. 


Operationalizing Listening in Deliberative Democracy 


According to the existing empirical literature on deliberative democracy, there are 
three main ways to measure listening: listening as exposure, listening as responsive- 
ness, and listening as deliberative effects. In this section, I offer an overview of these 
measures and highlight some of their limitations. As I will show, none of these mea- 
sures, on their own, can adequately capture all the normatively relevant aspects of 
listening itself. In other words, while some of these efforts to operationalize listening 
go too far in equating listening with its outcomes, others do not go far enough and 
conflate listening with simply the opportunity to hear. 

Listening as Exposure. One way to operationalize listening is to link it to exposure, 
or to being in the presence of another who speaks or signs a comment in a language 
they can understand (Mutz 2006; Neblo et al. 2018). But while exposure to another 
person's input may be a necessary condition for listening, it is not sufficient. Exposure 
to an utterance should be seen as an opportunity for others to listen rather than as 
evidence of their actually having done so. 

Listening as Responsiveness. Responsiveness, or a verbal acknowledgement of what 
another says, is a common measure for listening (Steiner 2012, 269; Button and 
Garrett 2016, 42) and related concepts such as reciprocity, interactivity, and reflec- 
tion (Pedrini 2014; Himmelroos and Christensen 2014). According to the Discourse 
Quality Index (DQI) employed by Steiner (2012, 269), the presence of an ‘undis- 
torted’ response shows that a person listened to another. If a person’s response is a 
non-sequitur or clearly distorts what the other person said, then we would conclude 
that the person did not listen well. Similarly, the absence of a response is seen as evi- 
dence that a particular person ignored the speaker. Notably, the relevant item in the 
DQI is ‘respect (listening)’ (Steiner 2012, 269), and in some applications of the DQI, 
investigators look specifically for a respectful response (Pedrini 2014). 

While using responsiveness to assess listening has its advantages—especially in 
allowing researchers to assess listening quality after the fact through transcripts—it 
also has severe limitations. By equating listening with responsiveness (and respect 
for that matter), we miss instances of democratically significant listening that—for 
a variety of valid reasons—may not result in a response (Scudder 2020, 136). For 
example, a failure to respond might actually indicate careful listening whereby a per- 
son was convinced and no longer feels the need to respond or perhaps was offended 
and does not want to dignify a comment with a response. Alternatively, a listener 
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might not have the opportunity to respond if others are dominating the conversation. 
Ultimately, a lack of response is not reliable evidence of failed listening, and so this 
measure likely underestimates the amount of listening going on in a wide range of 
deliberative encounters. 

Listening as Deliberative Effects. Another way to get a sense of whether listening 
has occurred is to look for changes in deliberators’ behaviours and positions. Was a 
person persuaded or influenced by what she heard in deliberation (Gerber et al. 2014; 
Neblo et al. 2018, 131; Beauvais 2021)? Or if they were not persuaded, did listeners 
better integrate others’ perspectives into the justification they gave for their pref- 
erences (Niemeyer 2019)? While listening is likely a necessary condition for these 
effects, we cannot conclude that listening failed simply by their absence. 

A less common approach, however, is to look for effects of listening on the speaker 
instead of the listener. Political theorist Susan Bickford (1996) has suggested speaker 
satisfaction as one possible indicator of listening. As Bickford (1996, 157) explains, 
“Being listened to” is an experience we have in the world, whether or not we can 
point to an unambiguous indication of listening’ Incorporating speaker satisfaction 
into a measure of listening helps us get at important questions of sincerity which are 
notoriously tricky to assess (Bachtiger and Parkinson 2019, 61; Srader 2015). Still, 
participant perspectives are not always reliable. A person might interpret others’ con- 
tinued disagreement as a sign that they did not listen when, in reality, his or her 
arguments simply failed to convince. While we ought to avoid relying solely on sub- 
jective feelings of being heard when assessing democratic listening in deliberation, 
the LQI productively combines this measure with other, more intersubjectively valid 
indicators. 

Each of these measures touches on an aspect of listening. What matters for lis- 
tening, however, is usually the combination of several of these aspects. The LQI 
combines these and other necessarily incomplete measures of listening into a sin- 
gle scale, giving researchers a more accurate instrument for assessing listening in 
deliberative forums, such as citizens’ assemblies. 


Listening Quality Index 


The LQI offers a novel approach to measuring listening by combining elements of 
existing measures into a lexical scale.” The term ‘lexical scale’ comes from John 
Rawls’s lexical ordering of his three principles of justice (Rawls 1971, 42; Gerring et al. 
2021, 783). In A Theory of Justice, Rawls presents three principles of justice and then 
orders them according to ‘lexical priority.’ By this, Rawls means that the principles 


> For general discussions of lexical scales, see Rawls (1971), Neblo (2009), and Gerring et al. (2021). 

ĉ Rawls actually refers to two principles, with the second principle having two parts, that is, 1, 2a, and 
2b. These are the Liberty Principle (1), the Fair Equality of Opportunity Principle (2a), and the Difference 
Principle (2b). With his lexical ordering of these principles, Rawls is saying that, when pursuing justice, 
we must not trade basic liberties for more economic opportunity. 
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of justice are not interchangeable and cannot be summed. As such, satisfying two 
principles of justice is not always better than satisfying one. Instead, the order mat- 
ters. Given the principles’ lexical ordering, a society that satisfied the first principle 
of justice but not the others should still be seen as more just than a society that sat- 
isfied both the second and third principles, but not the first. According to Rawls, the 
second and third principles only carry their full normative weight in the presence of 
the first principle. And we should not consider principles 2 or 3 until principle 1 has 
been fully satisfied. 

As with Rawls’s principles of justice, the listening attributes I include in the scale 
below are ordered in a normative sense, where the lower-level items on the scale are 
more central to listening than each succeeding item. More specifically, the LQI is con- 
structed using the strategy of lexical scaling outlined by Gerring et al. in their 2021 
article.* As the authors explain, lexical scales are ordered deductively, using ‘a cumu- 
lative logic to aggregate attributes according to their logical entailments, functional 
dependence, and conceptual centrality’ (Gerring et al. 2021, 781). Their ordering, 
while contestable, ‘is a preempirical question’ based on our normative understanding 
of democratic listening (Gerring et al. 2021, 780). 

The LQI considers a discussion or deliberative encounter as an opportunity for all 
present to listen. The unit of analysis is the listener-to-speaker pairing (or directed 
dyad) at the conversation level.” Unlike with the DQI, where a person is only included 
in the dataset if they speak, the LQI gives all participants (both speakers and non- 
speakers) a listening score. The LQI’s data structure resembles a complete network 
analysis of the people present during a deliberation. If there are four people in a group, 
and each person speaks once, then each person would have three listening scores. In 
other words, we would code listening for Persons B, C, and D with respect to Person 
A’s contributions to the conversation. The LQI includes the following indicators: 


1. No listening: The listener reports not listening to the speaker. 

2. Listener reports listening to the speaker: The listener reports listening to the 
speaker, but interrupts the speaker to the point of silencing. 

3. Listener does not silence the speaker: The listener does not interrupt to the point 
of silencing the speaker. The listener allows the speaker to speak. 

4. Listener recalls speakers comment: The listener can recall what the speaker said. 

5. Listener responds to the speaker: The listener responds to the speaker. The 
response can be verbal or nonverbal (nodding, shaking head). 


* In the social sciences, lexical scales are often associated with ‘Guttman-type scales’ and ‘item response 
theory (Guttman 1950). The LQI, however, is a non-Guttman lexical scale. For more on this and the 
construction of the LQI, see Scudder (2022). 

5 As Mendonga et al. (2022) argue, we ought to ‘expand our understanding of deliberative democracy 
and make it more attuned to the non-verbal forms of communication (2). This is especially important 
when we talk about ‘macro’ deliberation occurring in the broader public sphere. The LQI, however, 
is designed for ‘micro’ deliberation with the aim of helping us better understand the extent to which 
participants listen to the verbal expressions of others. 
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6. Listener gives a substantive response to the speaker: The listener responds to the 
speaker in a substantive and relevant way. 

7. Speaker reports feeling heard by the listener: The speaker reports being satisfied 
with the sincerity of the listener’s listening. 


As these listening indicators are lexically ordered, for a person to receive a given score, 
all prior criteria must be met. Importantly, however, satisfying a higher-order crite- 
rion does not imply that the listener has also satisfied all lower-order criteria (Gerring 
et al. 2021, 800). In other words, it is possible for a person to satisfy criterion 4, but 
fail to satisfy criterion 3. This person would then receive a score of 2 according to the 
LQL A listener only moves up the scale if they satisfy all previous criteria. Thus, in 
order to receive a score of 6, a listener must satisfy criteria 1-6. For a visual represen- 
tation, see Figure 8.1. The names of each level of listening (listed in the last column 
in Figure 8.1) are intended only to describe these theoretically derived levels of lis- 
tening. In the remainder of this section, I elaborate on the significance of each of the 
listening indicators included in the LQI. A more detailed account of the instrument 
can be found in Figure 8.2 in Appendix 8.1 of this chapter. 


1. Listener reports listening to the speaker. Does the listener report having listened 
to the speaker? In political science and communication research, a self-report 
from would-be listeners has been used to capture levels of listening in conver- 
sations (Johnson et al. 2019, 2176; Eveland et al. 2020; Rojas 2008). When it 
comes to the LQI, in order to achieve above a 0, the listener must report having 
listened. This is a bare minimum. Ifa participant reports not having listened, we 
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Figure 8.1 Listening Quality Index 
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should trust this assessment. This first criterion gives us relevant information 
with which to interpret other behaviours. For example, a person who reports 
not listening should not get ‘credit, so to speak, for allowing another person 
to speak. In this case, not interrupting is consistent with the listener ‘check- 
ing out’ of the conversation. On the other hand, just because someone reports 
having listened, it does not mean that they have done so successfully or in a 
meaningful way. Thus, this self-report should be interpreted as an intent to lis- 
ten. It is a precondition for the other listening attributes to have any meaning 
whatsoever. Put differently, the absence of this attribute is more telling than its 
presence. A person reporting not having listened would receive a score of 0. 

2. Listener does not silence the speaker. Does the listener allow the speaker to 
speak? Or does the listener interrupt the speaker to the point of silencing 
them? Some level of interruption is consistent with listening, for example, 
interrupting to ask a clarifying question. The key element of this indicator 
relates to whether or not the speaker was interrupted to the point of being 
silenced. To meet this criterion, either the listener must not interrupt or else 
their interruption must not derail the speaker. 

3. Listener recalls the speakers comment. Can the listener recall what the speaker 
said? Simply being able to recall a person’s input does not prove that you lis- 
tened sincerely and with the intention of fairly considering what you heard 
(Mueller and Oppenheimer 2014). But recall does help establish that the lis- 
tening act met a minimum threshold of attentiveness (Sillars et al. 1990, 505). 
Most often used in measures of listening comprehension, recall is a relatively 
noisy measure. It captures not only how well someone listened, but also the 
listener’s memory, intelligence, and his or her prior knowledge of a subject. 
Successful recall could also be driven by the speaker’s style of speaking. More- 
over, it is important to acknowledge that even the most engaged and generous 
listener will not always be able to recall the specifics of what someone said, 
especially if recall is measured only after deliberation ends. 

Importantly, by placing recall after both self-reports of listening and letting 
someone speak, we are able to capture instances of listening (types 1 and 2) even 
when it is not ‘successful’ in the sense of effectively transmitting a message. 
Type 3, or what I describe as ‘attentive listening, would occur only when the 
listener reports having listened, does not interrupt the speaker, and can recall 
a particular message communicated in a listening opportunity.° 

Existing measures of listening as responsiveness fail to register, let alone dif- 
ferentiate between, any of the categories of listening described so far. None of 
the preceding attributes would be registered using the DQI because they do not 


é The purpose of including an independent recall measure in the LQI is to give listeners the opportunity 
to demonstrate listening in the absence of a response. Still, given the nature of recall and how the measure 
works (being asked only after deliberation is over and being susceptible to measurement error because of 
this delay), it makes sense to allow a substantive response (criterion 5), when present, to satisfy the third 
criterion. 
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produce a speech act to be assessed. By incorporating these first three criteria 
into the measure, we are able to look for instances of more passive listening 
on the part of those who either do not want to speak up or do not have the 
opportunity to do so. 

4. Listener responds to the speaker. Does the listener provide any sort of response to 
the speaker? This next level of the LQI is achieved when the listener provides 
at least a simple response to the speaker. This category includes non-verbal 
responses such as a nod or shake of the head, or a simple question like ‘Can 
you repeat that?’ This kind of response should only be considered as evidence 
of listening if all previous criteria have been satisfied. That is to say, for a simple 
response to provide an indication of someone having listened, the listener must 
also report having listened, let the other person speak, and be able to recall what 
was said. 

5. Listener gives a substantive response to the speaker. Does the listener respond 
substantively to the speaker?’ A substantive response would be any form of 
acknowledgement of what the speaker has said. A simple ‘okay’ would not qual- 
ify. This response can be in the form of a question, including a substantive or 
rhetorical question, including ‘what did you mean by X? To satisfy this indica- 
tor, the listener might ask the speaker to expand on what he said or to explain 
how it relates to something else. 

This item of the LQI corresponds most closely to the DQI’s measure of 
‘respect (listening)’ and is satisfied when the listener does not ignore a speaker's 
comment. For the LQI, however, a substantive response need not communicate 
respect for what was heard (Pedrini 2014; Pedrini et al. 2013). Even a dismissive 
response (e.g. Tm not going to listen to this’) indicates some level of listening, 
and thus would satisfy this condition. 

Importantly, however, to qualify as a true response, it must relate to what 
the other person has said. The particularities of what the speaker said must be 
taken into account in the response. A non-sequitur, whereby someone changes 
the subject would not satisfy this criterion, no matter how substantive it was. 
As with the DQI, this will be judged by the investigator or other third-party 
coders. 

6. Speaker reports being heard by the listener. Does the speaker report feeling 
heard by the listener? Most of us have a sense of when someone is really 
paying attention to us versus merely going through the motions or com- 
ing up with a response (Bickford 1996, 157). In a small group experimental 
setting, this would entail asking individuals to report the extent to which 
they felt heard by particular others. Speakers’ perceptions are a valuable 


7 Note, a person who gives a substantive response (criterion 5) should also be seen as having satisfied 
the simple response indicator (criterion 4). This has implications for data collection and analysis: even if 
listeners do not nod their head or ask a speaker to repeat herself, if they respond in a substantive way then 
we would take them to have satisfied both criteria 4 and 5. 
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indicator of listening as they help get at issues of sincerity and authentic- 
ity that have long plagued empirical approaches to the study of deliberation 
(Ercan with Bachtiger 2019, 102). 

Given the placement of this indicator in the lexical scale, we would only con- 
sider a speaker’s perception of being heard if the other components of listening 
are also satisfied. In other words, the speaker’s satisfaction would enhance the 
overall listening score, but the absence would not detract from other indica- 
tors of listening, including a substantive response. Drawing on the ‘listening 
act theory’ I present elsewhere (Scudder 2020, 87-105), I describe this level 
as ‘performative listening: Here, we judge the listening to have met the per- 
formative threshold needed to achieve a meaningful level of intersubjectivity 
or engagement with others. Importantly, by performative listening, I do not 
mean listening that is staged or affected. Instead, I mean listening that is done 
for listening’s sake rather than for strategic reasons.® 

As mentioned above, questions of sincerity are notoriously difficult to get at 
empirically. Still, this category offers the most evidence that sincere listening 
has occurred: the listener reports having listened, the third-party coders find 
independent evidence of the listener having listened, and the speaker reports 
feeling heard. In other words, all signs point to a meaningful level of listening 
having been achieved. 


Employing as it does the strategy of lexical scaling, the LQI solves the problem 
of aggregation, allowing us to make quality comparisons between various observed 
behaviours. For example, using the LQI, we are able to compare a level 2 case wherein 
a person did not interrupt, but was unable to recall what a speaker said, and a level 1 
case where a person replied, but without first letting a speaker finish her thought. 

The data required for the LQI is generated both during and after the delibera- 
tive encounter, using participant surveys or interviews and investigator observations. 
Specifically, observations for indicators 1 (listener self-report), 3 (recall), and 6 
(speaker self-report) are collected through participant surveys conducted as soon 
as possible after the deliberation concludes. Indicators 2 (interruption), 4 (sim- 
ple response), and 5 (substantive response) are measured by the investigator or a 
third-party coder who observes the deliberation. For these investigator-coded indi- 
cators, the deliberation needs to be either observed in real time or else recorded for 
post-deliberation evaluation. 

Because the LQI relies on participant reports in addition to investigator obser- 
vations, it cannot be applied easily to cases of deliberation after the fact. The LQI 
is designed for use in real time and does not allow for researchers to assess listening 
solely on the use of transcripts or records of previously occurring deliberation. While 


ê For more on this usage of ‘performative, see Austin (1962), Habermas (1984), and Srader (2015). 
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the inclusion of real-time reports from participants improves the instrument, it also 
makes it less widely applicable. 

In sum, by combining various indicators of listening, the LQI offers a more com- 
plete and normatively relevant measure of listening in democratic deliberation. And 
as I will explain below, this measure is well suited for identifying both the antecedents 
and effects of quality listening. 


The LQI in Practice 


Listening carries both normative and practical significance for deliberation. As such, 
the LQI can help us answer important questions, especially as they pertain to micro or 
small-scale deliberation. For example, deliberative democracy research reveals that 
participation in deliberative forums ‘facilitates political learning, promotes individual 
opinion change and increases subjective political efficacy’ (Suiter et al. 2016, 199). 
Future research using the LQI might examine whether these effects are mediated by 
listening quality. 

For example, recent work aiming to understand opinion change amongst partici- 
pants in the Irish We the Citizens deliberative experiment found that the ‘deliberative 
citizen, or the ‘citizen most likely to shift opinion following deliberation, is under 65, 
with median levels of knowledge’ (Suiter et al. 2016, 198). For Suiter et al. (2016) par- 
ticipation, as such, was the key experimental treatment, though they also looked at 
group composition, for example, how heterogeneous a group was. Importantly, how- 
ever, in studying the dynamics of opinion change, the investigators did not account 
for differing levels of participation amongst citizens. Building on this research, future 
work using the LQI could examine whether and how the listening capacities of 
citizens’ assembly participants impact the outcomes of deliberation. Do more ‘delib- 
erative citizens’ listen more or better than their less deliberative counterparts? If we 
found, for example, that those most likely to change their opinion also tended to be 
better listeners, then we would know where to focus our efforts to improve citizens’ 
deliberative capacity. 

Alternatively, the LQI could help us assess whether listening is impacted by the 
particular issue or topic being discussed. For example, does listening quality increase 
when citizens are discussing less salient issues, compared to periods when they are 
discussing more hot-button issues? How does this potential variation in listening 
affect changes in opinion? Besides citizens’ assemblies, the LQI can also be used 
to better understand the dynamics of deliberative polling (Fishkin 2018) and the 
deliberative town halls designed by Neblo et al. (2018). 

In these contexts and more, a cumulative LQI score can be used as either a depen- 
dent or independent variable. Using an LQI score as a dependent variable, we can 
investigate the factors that bear on listening quality. How do different deliberative 
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formats (e.g. moderated or not, small groups, big groups, homogenous or hetero- 
geneous groups, online or face-to-face) impact listening quality? We can also study 
whether some groups of people listen more or better than others, and whether 
actor-specific factors, like gender or age, affect listening quality. Are some people 
more likely to be listened to than others (Beauvais 2021)? In other words, the LQI 
can be used to identify the factors that facilitate or disrupt democratic listening. 

As an independent variable, on the other hand, an LQI score can help uncover the 
effects of high- or low-quality listening in either experimental settings or deliberative 
forums like the Irish citizens’ assembly discussed above. Does listening or being lis- 
tened to affect a person’s respect for her fellow citizens or representatives? Does the 
quality of listening in a decision-making process increase citizens’ perceptions of the 
legitimacy of the policy outcome? In sum, researchers can use participants’ compos- 
ite LQI scores to better capture both the antecedents and outcomes associated with 
listening quality. 

Furthermore, the LQI’s flexibility allows for disaggregation. Some research ques- 
tions might not require the entire index or aggregate score. In such cases, elements of 
this index, or particular listening indicators, could be peeled off to study particular 
relationships between listening behaviours and deliberative outcomes. For example, 
we might want to study whether certain deliberative conditions, moderated inter- 
actions for example, are more conducive to avoiding interruptions or generating a 
response. The LQI allows for some elements to be used in the absence of others when 
‘listening quality’ as a whole is not the particular subject of interest. 


Conclusion 


The LQI offers a new way to measure and assess the quality of listening amongst par- 
ticipants deliberating in small-scale deliberative venues or mini-publics. Improving 
on existing measures of listening in micro deliberation, the LQI helps us capture lis- 
tening quality beyond mere exposure but without fully collapsing the categories of 
listening and responsiveness. As such, the LQI conceives of listening as a distinct 
practice of deliberation and helps to push empirical deliberative democracy research 
beyond a politics of voice. That said, LQI is still a relatively voice-centric measure of 
listening, insofar as it focuses on listening to speech acts. In other words, in its current 
form, the LQI cannot easily capture listening to non-verbal or embodied expression, 
nor to what is not said (Mendonga et al. 2022). 

While the LQI can help improve our understanding of listening within discrete 
moments of deliberation, deliberative democracy is clearly broader than the mini- 
publics and small-scale deliberation discussed here (Chambers 2009; Hendriks 2006; 
Scudder 2021). Deliberation also occurs on a systems level (Mansbridge et al. 2012). 
And the democratic quality of this system-wide deliberation is dependent on the 
quality of listening taking place amongst and between citizens and their represen- 
tatives (Scudder 2020). As such, we must also consider how to measure and assess 
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the listening that occurs (or not) within and across broad and unstructured public 
spheres. As Bachtiger and Parkinson (2019) suggest, the tools we use to assess micro 
deliberation are not always applicable to macro deliberation. Still, the LQI provides 
a useful starting point to think about system-wide listening: teaching us that quality 
listening should not be confused with mere exposure to what someone says, while also 
reminding us that meaningful listening can occur in the absence of obvious policy 
effects. 
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discussion, how often did 
you (Person A) listen 
carefully to what Person X 
had to say? 


Did the subject (Person A) 
ever interrupt Person X? 
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conversation as you 
(Person A) can recall. 


Did the subject (Person A) 
show any sign of response 
to Person X. 


Did the subject (Person A) 
offer a substantive 
response to Person X? 
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* This score indicates that the subject (Person A) has satisfied the criteria for this level. As such, this participant's cumulative LQI score would be at least 
1. The coders would only move on to assess the participant on the second indicator (interruption) if he/she answered either (3), (4), or (5) for this first 
question. The same logic applies throughout the Table. A participant whose response to the third question (recall) is coded as two would receive an 
aggregate LQI score of at least three, and only these subjects would move on to be assessed on the ‘simple response’ indicator. 


Figure 8.2 LQI Measurement Instrument 
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Macro-level Assessment of Deliberative 
Quality 


Dannica Fleuß 


Deliberative theory stipulates that deliberative democracy has a positive impact on 
citizens’ political attitudes and competencies and may help to bridge deep disagree- 
ments and inner-societal divides (Dryzek et al. 2019; Ercan 2017; Mutz 2006). But are 
citizens in ‘more deliberative democracies actually more engaged, open-minded, and 
satisfied with political institutions and outputs? Are more deliberative democracies 
really better equipped to counteract populist challenges? And what concrete institu- 
tional reforms can facilitate and promote democracies’ deliberativeness? Questions 
like these call for systematic comparative analyses of democratic systems’ deliber- 
ativeness (henceforth: ‘macro-deliberativeness’). Yet established measurements of 
democratic quality cannot serve as a basis for comparative political scientists to 
answer these timely questions: they usually depart from distinctly liberal concep- 
tualizations of democratic quality that give little attention to the quality of political 
communication within and across different social and political spheres. 

This chapter introduces the method of macro-level assessment, which is devel- 
oped to conduct such analyses and determine the degree to which a democracy 
fulfils the normative requirements of deliberative theory. So far, the most advanced 
attempt at providing an index of nation-states’ deliberative quality is provided by the 
Varieties of Democracy (V-Dem) (Coppedge et al. 2020a, 2020b). Despite its var- 
ious strengths, however, this measurement approach cannot live up to its promise 
to provide a theoretically sound and methodologically robust measurement. V-Dem 
bases its assessment on criteria that are frequently used to evaluate deliberative proce- 
dures in individual forums (e.g. Steenbergen et al. 2003). The core approach involves 
experts evaluating deliberation ‘at all levels of society’ based on their level of jus- 
tification, common-good orientation, respectfulness, and inclusiveness (for a more 
detailed account, see Fleuf and Helbig 2021, 315). Yet a closer look at this approach 
reveals that a measurement of macro-deliberativeness must not simply ‘transfer 
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criteria that were applicable at the micro/meso level to a larger scale, but should 
account for the specifics of deliberativeness at a systemic level in conceptualiza- 
tion, operationalization, and measurement (Fleuf et al. 2018, 12; see also Fleuß and 
Helbig 2021). 

Any attempt to conduct a valid measurement of macro-deliberativeness that meets 
the standards of both deliberative theory and comparative political science must 
face a range of conceptual, methodological, and research-pragmatic challenges. This 
chapter demonstrates that measuring macro-deliberativeness is demanding, but that 
these challenges can be met by joining two strands of research, that is, the theoretical 
underpinnings of systemic deliberative theory and the procedural and method- 
ological guidelines of democracy measurements. The first section of this chapter 
outlines step-by-step guidelines for conceptualizing, operationalizing, measuring, 
and aggregating macro-deliberativeness. Although until now there has been no 
one-size-fits-all solution for measuring macro-deliberativeness (Dryzek 2009, 1390; 
Fleuf et al. 2018), scholars can make good use of existing methodological tools in 
deliberative democracy. Against this background, the second section illustrates by 
means of exemplary applications how researchers can utilize and combine existing 
methodological tools to measure macro-deliberativeness. The chapter concludes with 
recommendations for applying this ‘methodological toolkit’ to comparative studies 
seeking to assess the deliberativeness of different democracies. 


Three Phases of Measuring Macro-Deliberativeness 


The process of measuring democratic quality is usually subdivided into three phases 
that pose distinct methodological and research-pragmatic challenges. Comparative 
politics scholars have developed strategies for meeting these challenges in measur- 
ing democratic quality (Munck and Verkuilen 2002; see also Moller and Skaaning 
2012; Pickel et al. 2015). The following paragraphs apply these insights and provide 
strategies for conceptualizing (Phase 1) and operationalizing macro-deliberativeness 
(Phase 2) as well as for providing an aggregate measure or ‘score’ that represents a 
democratic systems overall deliberative quality (Phase 3). 


Phase 1: Conceptualization 


All measurements of democracy start out from a conceptualization that speci- 
fies evaluation criteria. Unsurprisingly, the concepts ‘democracy’ and ‘democratic 
quality’ are notoriously subject to disagreement, but conceptualizations applied in 
established measurements share one feature: they almost exclusively refer to liberal 
understandings of democracy. Accordingly, they operationalize democratic quality 
mainly by indicators measuring the quality of elections, their competitiveness, as well 
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as (constitutionalized) checks and balances and individual liberties (e.g. Buhlmann 
et al. 2012, 522). 

Deliberative understandings of democratic quality, meanwhile, highlight the value 
of reasoned, inclusive, respectful, coercion-free communicative exchanges that take 
place in public and empowered spaces. Collectively binding decisions—democratic 
laws and policies—must be legitimated ‘bottom-up; that is, by deliberation in the 
public sphere and broader civil society. Therefore, a high-quality democracy needs 
deliberation in different ‘sites’ or ‘spaces’ of political-societal systems and ‘flows in 
communication’ or transmission between public and empowered spaces (see Dryzek 
and Niemeyer 2010, 11; Habermas 1996). Systemic deliberative theorists conceptu- 
alize macro-deliberativeness as an ‘emergent property’ that ‘cannot be reduced to a 
mere aggregation of other qualities of the political system’—that is, to an accumu- 
lation of individual procedures’ deliberativeness (Fleuf et al. 2018, 12; Mansbridge 
et al. 2012, 1). 

A core challenge of measuring macro-deliberativeness is developing a sufficiently 
precise conceptual account of this ‘fluid’ or ‘emergent’ phenomenon. Scholars must 
identify the concepts’ core dimensions (their ‘attributes’), define their characteristics 
and outline their relationships (Pickel et al. 2015, 504). The most practical way to 
represent this conceptual logic is to construct a ‘concept tree. Concept trees orga- 
nize attributes of complex concepts vertically by level of abstraction; the elements 
at the lowest level of abstraction are metaphorically termed ‘leaves.’ These concept 
trees must avoid being too minimalist or too maximalist: Researchers should aim 
at including all attributes necessary to differentiate ‘macro-deliberativeness’ from 
related concepts—and they should include only attributes necessary to discrimi- 
nate amongst cases studied (Munck and Verkuilen 2002, 9, 21). The concept tree 
for macro-deliberativeness that identifies and organizes the concepts attributes and 
sub-attributes should reflect two fundamental premises of systemic deliberative the- 
ories: Deliberative democracies are characterized by (a) deliberation in different sys- 
temic sites, and (b) communicative exchanges—transmissions—between these sites 
(see Figure 9.1). 

Undoubtedly, this account still leaves some room for interpretation: Researchers 
must (a) further determine what evaluation standards should be applied to evalu- 
ate deliberation in individual deliberative sites (that is, what ‘meso-deliberativeness’ 
means). For the sake of simplicity, we can organize the existing conceptual diver- 
sity by distinguishing two ideal types. Both types mainly differ in how broadly they 
conceptualize ‘deliberation’: While narrow type I conceptualizations refer to strictly 
rational, consensus-oriented reason-giving, more expansive type II conceptualiza- 
tions incorporate ‘alternative forms of communication (Bachtiger et al. 2010, 33-34). 
Assessing deliberation at a systemic level necessarily involves studying heterogeneous 


* In constructing this concept tree, scholars should respect two ‘basic rules of conceptual logic’: To avoid 
conflation, attributes must be organized according to their level of abstraction and assorted to the ‘proper 
branch of the concept tree. To avoid redundancy, attributes at the same level of abstraction should not 
‘overlap but refer to ‘mutually exclusive aspects’ (Munck and Verkuilen 2002, 13). 
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Figure 9.1 Concept Tree 
Source: Author’s own graph. 


deliberative procedures in diverse sites. Against this backdrop, scholars have recom- 
mended including ‘different styles or forms of reason-giving’ in systemic assessments 
(Esau et al. 2020, 90). There may be no definite recommendation as to include specific 
alternative forms of communication such as rhetoric communication or narratives. 
Yet, particularly when researchers wish to include an assessment of political commu- 
nications in informal settings, it has proven useful to consider emotional expressions 
and narratives as potential means of deliberation (see Esau et al. 2020). 

Asa conceptual basis for measuring deliberativeness at a systemic level, researchers 
should provide a ‘map’ of the deliberative sites to be assessed. Minimally, this map 
should include a differentiation between empowered spaces, such as parliaments or 
national courts that are home to deliberation with the capacity to make collectively 
binding decisions, and public spaces such as the mass media or civil society organiza- 
tions that are the locus of opinion-formation processes. As deliberative sites in public 
and empowered spaces fulfil distinct functions for democratic processes, researchers 
should differentiate them in their conceptual framework and assess them separately 
(Bachtiger and Parkinson 2019). 

This ‘map’ of deliberative sites is also crucial for the conceptualization (and 
the consecutive measurement and aggregation) of the second attribute of the con- 
cept tree: to specify (b) ‘transmissions, researchers need to be clear about what 
‘communicative flows’ between which sites they consider relevant for democracies’ 
macro-deliberativeness. At a fairly abstract level, transmissions between delibera- 
tive sites can be characterized as a triple relationship: Arguments or discourses are 
transmitted from public to empowered sites of a democratic system (or vice versa). 
Although transmissions cannot be conceptualized as ‘a one way-process’ (see Neblo 
2005, 177-178), deliberative democrats tend to put prime value on ‘bottom-up trans- 
missions that proliferate arguments and discourses from deliberations in the public 
to deliberations in the empowered space (Dryzek and Niemeyer 2010, 11). 


Phase 2: Operationalization and Measurement 


In Phase 2, researchers depart from the ‘attributes at the lowest level of 
abstraction which they identified by constructing the concept tree (Munck and 
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Verkuilen 2002, 15). ‘[MJeasurement processes build numerical bridges between 
abstract concepts and empirical realities’ (Schedler 2012, 22; Pickel et al. 2015, 
506). Consequently, the core task here is to assort the (sub-)attributes of macro- 
deliberativeness which refer to empirically observable phenomena and facilitate a 
valid, reliable, and replicable measurement. 

To be valid, indicators must correctly and comprehensively measure the abstract 
concept. Multiple indicators reduce measurement error and accommodate for the 
fact that there may be multiple manifestations of one conceptual attribute in empir- 
ical reality (Munck and Verkuilen 2002, 15-16). The process of measuring complex 
concepts ‘requires the specification of the rules by which numerals are assigned to 
objects. Hence, the validity of indicators also depends on the measurement level 
(Pickel et al. 2015, 507). While there are no ‘a priori grounds’ for choosing a particu- 
lar measurement level, researchers should avoid choosing scale levels that introduce 
excessively fine-grained or coarse-grained distinctions (Munck and Verkuilen 2002, 
17). There is, however, no hard and fast rule to choosing either appropriate indi- 
cators or measurement levels. Rather, these steps of developing a measurement 
approach require ‘both theoretical justification and empirical testing’ (Fleuß and 
Helbig 2021, 310). 

Due to the talk-centric nature of deliberative understandings of democratic qual- 
ity, the measurement of the two attributes (a) deliberation in individual sites and 
(b) transmissions (see Figure 9.1) must refer to textual data from political com- 
munications in public and empowered spaces. Consequently, a measurement of 
macro-deliberativeness cannot rely on ‘hard data’ but must at least be partly based on 
‘subjective’ data, that is (manual) coding procedures. Here, it is crucial that the guide- 
lines for coders are precise and unambiguous (such that different coders interpret 
them in the same way and the measurement’s reliability is ensured) and transparently 
documented (to ensure replicability) (Pickel et al. 2015, 507). 


Phase 3: Aggregation 


Measuring democracies’ deliberativeness requires not only measurement approaches 
for (a) deliberation in individual sites and (b) transmission, but also demands a 
strategy for aggregating individual (sub-)attributes to a score that represents democ- 
racie’ macro-deliberativeness. In a nutshell, in Phase 3 researchers must ‘reverse 
the process of disaggregation that was carried out during the conceptualization 
[and operationalization] phase. The fundamental benchmark that researchers must 
meet when aggregating attributes of complex concepts is ‘concept-measurement con- 
sistency’ (Goertz 2006). To determine an aggregation strategy, researchers must 
therefore be clear about the logical relationships between the concepts’ attributes and 
sub-attributes (as represented in the concept tree). Aggregation procedures usually 
involve multiple rules: They start out with rules for aggregating the numerical values 
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Figure 9.2 Relationships between Attributes 
Source: Author’s own graph. 


assigned to the leaves of the concept tree and move step by step to higher levels of 
abstraction.” 

I focus on two questions to illustrate the choices that researchers have to 
make here in assessing macro-deliberativeness (see Figure 9.2): First, are its 
attributes and/or sub-attributes necessary conditions for democracies’ macro- 
deliberativeness? Second, do they interact with each other, that is, is the score of 
one (sub-)attribute impacted or conditioned by the score of another (sub-)attribute 
(see Moller and Skaaning 2012, 236)? 

The conceptualization of macro-deliberativeness outlined above leaves room for 
interpretation. Yet this basic conceptual account determines core parameters of suit- 
able aggregation strategies (see Figure 9.2): First, systemic theories of deliberation 
assume a ‘division of labour’ between different deliberative sites and thereby pre- 
suppose that deliberations can, at least to some extent, compensate for the lack 
thereof in other sites (Fleuf et al. 2018, 18). Second, deliberative democracies require 
deliberation in individual sites and transmissions between sites (i.e. both attributes 
are necessary and non-compensatory). To further specify suitable aggregation strate- 
gies, potential interactive effects between (sub-)attributes must be considered. While 
there is no indication to consider the scores for meso-deliberativeness in individ- 
ual systemic sites as interactive, it makes good sense to think about the scores for 
(a) deliberation in individual sites and (b) transmissions as mutually conditioned. 
The quantity and quality of deliberation in individual sites in the deliberative sys- 
tem may impact the extent to which discourses are transmitted across other sites. For 


> At a foundational level, this mathematical procedure includes two core components: operations and 
weights. Weights must reflect the attributes’ relative importance for ‘democratic quality. Operations mir- 
ror the attributes’ relationship, for example, if attributes are considered as necessary or substitutable 
elements of a concept. While the choice of aggregation procedures should be theory-driven, it also requires 
empirical testing and robustness checks (Munck and Verkuilen 2002, 8, 25; Moller and Skaaning 2012; see 
also Fleuß and Helbig 2021, 321). To ensure the measurements’ replicability, the aggregation rule must be 
documented and published (Pickel et al. 2015, 500). 
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Table 9.1 Aggregation Rules 


Attribute necessary or Interactive effects between 

Aggregation rule compensatory? attributes? 
Multiplication Necessary Interactive 

Minimum rule Necessary Noninteractive 
Geometric mean (Partially) compensatory Interactive 

Arithmetic mean or (Partially) compensatory Noninteractive 

addition 

Maximum rule Compensatory Noninteractive 


Source: Moller and Skaaning (2012, 236). 


example, numerous and intense debates in public spaces make it more likely to solicit 
reactions and engagement from actors in the empowered space. Likewise, ‘com- 
municative flows’ between different deliberative forums are likely to inspire more 
frequent, intense and constructive debates in individual forums. However, the inter- 
activity between attributes is not a purely theoretical-conceptual matter, but requires 
empirical testing to ‘gain a sense of the robustness of aggregate data (Munck and 
Verkuilen 2002, 25). 

These considerations suggest appropriate aggregation strategies for assessments of 
macro-deliberativeness (see Table 9.1): While the scores for meso-deliberativeness 
in different systemic sites can be aggregated with an additive rule, the stipulated 
relationship between the attributes (a) deliberation in individual sites and (b) trans- 
missions calls for a multiplicative procedure. 


A Methodological Toolkit for Measuring 
Macro-Deliberativeness 


There is no one-size-fits-all solution for measuring macro-deliberativeness (Dryzek 
2009, 1390; Fleuf et al. 2018). Yet scholars can use existing methodological tools if 
they apply and combine them in a manner that reflects both the normative bench- 
marks of deliberative theory and the methodological requirements of democracy 
measurements. In this section, I shall demonstrate how such tools can provide the 
means for assessing the core attributes (a) and (b) identified in the conceptualiza- 
tion phase (see Figure 91). By applying and combining them in accordance with the 
conceptual and methodological recommendations outlined above, researchers can 
then assess democracies’ macro-deliberativeness (see Figure 9.2). Due to the talk- 
centric nature of deliberative democracy, the unifying feature of these tools is that 
they rely on methods for analysing political communication, specifically quantitative 
text analysis methods. 
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Instruments for Measuring Meso-Deliberativeness 
in Individual Sites 


Deliberative democracy scholarship has developed a broad variety of well-tested 
approaches for measuring meso-deliberativeness (for an overview of direct and indi- 
rect measures, see Black et al. 2010). For illustrative purposes, this section focuses 
on guidelines for applying the Discourse Quality Index (DQI) (Steenbergen et al. 
2003), which is often considered as the ‘best practice example’ for meso-level assess- 
ments (see Fleuf et al. 2018, 11; see also Bachtiger, Gerber, and Fournier-Tombs, 
Chapter 6 in this volume). The DQI presents a theory-driven operationalization of 
meso-deliberativeness and corresponding coding instructions for evaluating political 
communication. Its original coding categories reflect the evaluation criteria as sug- 
gested by type I-concepts. They refer to speakers’ free and uncoerced participation, 
the level and content of justification, and the respectfulness and constructiveness of 
their contributions (Steenbergen et al. 2003, 27-30). 

The main challenge of using established measures of meso-deliberativeness in 
assessments at the systems level results from the fact that they must include het- 
erogeneous deliberative sites. The DQI’s measurement was originally developed to 
study parliamentary deliberation, that is, communication in highly formalized and 
regulated settings in the empowered space. Researchers who aim at a measurement 
of macro-deliberativeness must assess debates in empowered and public spaces. 
To analyse debates in diverse systemic sites, it can be necessary to adjust coding 
procedures and instructions (also see Esau et al. 2020). 

The following exemplary application of the DQI to German debates on abor- 
tion policies (1991-1993) is based on two data sources: First, on data generated in 
a research project that has been explicitly devoted to applying the DQI to German 
empowered space debates (see Bachtiger et al. 2005, 2008; Steiner et al. 2004). Second, 
the meso-deliberativeness of public space debates has been evaluated based on an 
analogous assessment of German mass media contributions; more specifically, ran- 
domly selected articles published between January 1991 and December 1993 in four 
leading national newspapers covering a broad political spectrum (for an overview of 
the results, see Figure 9.3 and Table 9.2).° 

A number of issues that are particularly relevant for macro-level assessments 
became visible in this exemplary analysis. For example, for the analysis of newspaper 
articles it was necessary to adjust the unit of analysis (in the analysis of parliamen- 
tary debates: individual speeches) as there is no ‘natural’ way to subdivide articles 
into individual ‘speeches: Further, details of the coding instructions used to evaluate 
debates’ respectfulness (particularly the DQI-indicator ‘respect towards demands’ 


* The text corpus for the analysis of public space debates’ meso-deliberativeness is identical to the corpus 
used in the transmission analysis (see note 6). I would like to thank André Bachtiger for providing me with 
the disaggregated data for empowered space debates. Here, the units of analysis are individual speeches 
(for both parliamentary debates N = 193; heckles excluded). 
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Figure 9.3 Meso-Deliberativeness Measures in Different Deliberative Sites 
Source: Author’s own graph. 


Table 9.2 Measurement of Meso-Deliberativeness in Public and Empowered Spaces 


Indicator jlevel rgroup rcount2 cp Meso-Deliberativeness 
Public Space (mass media debates) 


May 91-26 June 92 2.6 0.6 1.6 0.4 52 
—28 May 93 2.1 0.4 1.7 0.7 4.9 
-Dec 93 2.5 0.5 1.5 1 5.5 


Empowered Space (parliamentary debates) 
26 Sept 91 3.3 1:3 1 0.6 6.2 
25 June 92 2.9 1:3 1.1 0 53 


Source: Author’s own graph. 


which was originally used to evaluate parliamentary debates that explicitly aim at 
deciding upon the ‘demands’ put forward by different speakers) were not applicable. 
Given the controversial topic, the public space debates displayed surprisingly high 
levels of justification and high meso-deliberativeness scores (see Figure 9.3).* 
However, this is likely to also result from the kind of public sphere debates 
analysed: the text corpus was compiled from elite national newspapers. German 
(elite) newspapers privilege ‘[e]motionally detached, disembodied argumentation 
while considering ‘personal narratives [highly inappropriate]’ (Ferree et al. 2002, 
273). While this makes elite newspaper debates likely to conform with the original 
DQI’s measurement standards (Steenbergen et al. 2003), this need not be con- 
ducive to macro-deliberativeness, but rather may also have exclusionary effects: 
Especially ‘personal narratives’ in mass media debates can be ‘a way of helping 


* In line with Steenbergen et al’s (2003) original approach, the meso-deliberativeness scores here have 
been computed with the help of an additive aggregation rule (i.e. meso-deliberativeness = jlevel + rgroup 
+ rcount2 + cp). For alternative strategies and amendments, see Bachtiger et al., Chapter 6 in this volume. 
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citizens connect their own experiences with a policy debate and as a way of over- 
coming discourse that has become unreal through too much abstract argumentation’ 
(Ferree et al. 2002, 273). 

Particularly when informal conversations or ‘everyday political talk are included 
in the assessment, researchers may consider including additional indicators for alter- 
native forms of communication that capture, for example ‘whether a comment 
reported a personal experience [. . .] expressed in narrative forn or ‘contained pos- 
itive [or negative] emotions’ (Esau et al. 2020, 102).* Depending on the quantity of 
communications they want to include in their assessment of macro-deliberativeness, 
researchers should consider the potentials of conducting the analysis of meso- 
deliberativeness with the help of machine learning tools (e.g. Fournier-Tombs and 
Di Marzo Serugendo 2019). 


Instruments for Measuring Transmissions 


To assess transmission, researchers must essentially measure the quantity of argu- 
ments or discourses proliferated across systemic sites. This measure should be 
complemented by an analysis that tracks ‘where the transmitted elements come from 
and where they are transmitted to’ (Fleuf et al. 2018, 18). While measurements of 
meso-deliberativeness such as the DQI refer to the procedural qualities of politi- 
cal communication (Steenbergen et al. 2003, 43), measuring transmissions requires 
scholars to explore the content of political communications in diverse systemic sites. 
Case studies frequently apply interpretive research methods to determine such con- 
tent and apply methods such as process tracing (see Collier 2011; see also Pickering, 
Chapter 20 in this volume) to identify the mechanisms and conditions relevant to 
transmission (Ercan et al. 2017; Hendriks et al. 2007; see Papadopoulos 2012). 

The basic methodological strategy for measuring transmissions in democratic sys- 
tems can be summarized in four basic steps (see also Bachtiger and Parkinson 2019, 
chapter 7; Beste 2016; Parkinson et al. 2020): 


Step 1: Content exploration—Researchers should rely on an inductive approach 
to identifying the ‘discursive elements’ (arguments or topics) (Ercan et al. 
2017) in public and empowered spaces. 

Step 2: Identification of main discourses—They should then combine this con- 
tent exploration with clustering techniques that are either theory-driven 
or guided by secondary (qualitative) research. Clustering techniques 


° The heterogeneity of deliberations in various sites may also impact index-building procedures such 
that context-sensitive aggregation strategies for calculating meso-deliberativeness scores may be required 
(Bachtiger and Parkinson 2019, 62, 139). 
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group discursive elements and identify the fundamental overarching 
‘discourses’ (also see Niemeyer 2011, 107-108). 

Step 3: Analysis of spread of discourses—In the next step, the spread of these dis- 
courses across different sites of democratic systems should be tracked to 
answer the question: where did particular discourses occur at what point 
in time? 

Step 4: Transmission score—Finally, to provide a measure of communicative 
flows in a democratic system, researchers should calculate an aggregate 
score representing the quantity of transmissions between the public and 
empowered space. 


For the German abortion debate introduced above, manual coders inductively 
extracted fourteen main arguments from the corpus of sample public space 
debates (Step 1) and grouped them into four overarching discourses (Step 2).° 
Figure 9.3 shows the relative frequency with which arguments assorted to the 
discourses Women’s autonomy, States responsibility to protect unborn life, Relation- 
ship state/individual and social policy, and Comparison GDR occurred in sample 
empowered and public space debates (Step 3).’ 

To provide a proxy measure of transmissions between deliberative sites (Step 4), I 
first studied sets of subsequent debates in public and empowered spaces and analysed 
the extent to which respective discourses overlap (in each case, these sets included: 
public space debate 1—empowered space debate—succeeding public space debate 2, see 
Table 9.2).° I applied Gallaghers Least Squares Index (LSq) that is an established 
measure for electoral systems’ relative disproportionality between votes and parlia- 
mentary seats to calculate the extent to which discursive patterns between subsequent 
public and empowered space debates diverge: 


LSq = (05 x D (DiscPublic; — DiscEmpi)’ ) 


° The corpus for the explorative analysis was compiled from the two parliamentary debates held on 26 
September 1991 and 25 June 1991 and the federal court’s justification of its rule in May 1993 (https://www. 
servat.unibe.ch/dfr/bv088203.html#Opinion) as well as seventeen randomly selected articles published 
between January 1991 and December 1993 in four leading national newspapers covering a broad political 
spectrum (Frankfurter Allgemeine Zeitung, Stiddeutsche Zeitung, Der Spiegel, and Die Tageszeitung). The 
results of the analysis in Steps 1 and 2 were checked by two interdependently working coders for intercoder 
consistency (see Thomas and Harden 2008). 

7 The ‘women’s autonomy’ discourse is constituted out of a constellation of arguments which emphasize 
women’s sole responsibility to decide about their own body; ‘state’s responsibility to protect unborn life’ 
is comprised of arguments dealing with the state’s responsibility towards embryos and potential (legal) 
enforcements; the third discourse addresses the relationship between individual (women) and the state 
and related matters of social policy; and ‘Comparison GDR is comprised of arguments that concern the 
task of homogenizing the legal regulations on abortion in Eastern and Western Germany. 

ê This proxy measure of transmissions between two deliberative sites measures the ‘overlap’ of dis- 
courses. This overlap is certainly only a necessary, not a sufficient condition for successful transmissions 
of discourses between deliberative sites. 
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For democratic systems’ macro-deliberativeness, transmission from public to empow- 
ered spaces is of outstanding importance. Generally speaking, all communicative 
flows between public and empowered deliberative sites can be beneficial to demo- 
cratic systems’ quality. Yet when actors in the empowered space manage to impose 
their arguments or discourses on public spaces, this tends to negatively affect demo- 
cratic systems’ macro-deliberativeness. Accordingly, the rule used for calculating the 
Transmission Score (TS) should punish cases in which public space debates change 
subsequently to empowered space debates, such that they then mirror empowered 
space debates more closely than preceding public space debates. 

The following rule for calculating a transmission score constitutes one way to rep- 
resent this basic rationale. In this formula, LSq(A) represents the extent to which 
the discursive patterns in the first public space debate (public space debate 1) of 
the sequence analysed diverge from those in subsequent empowered space debates; 
LSq(B) is a measure for the divergence between patterns in empowered space and 
subsequent public debates (public debate 2). LSq(C) represents the extent to which 
public space debate 1 and public space debate 2 diverge.’ 


TS = [100 — 154 (A) a — LSq 2 x SO) 


In Sequence 1 (see Figure 9.3), the patterns of discourses between the first phase of 
public debates (May 1991 to June 1992) and subsequent empowered space debates 
(parliamentary debates in September 1991 and June 1992) display comparatively 
large deviations (LSq(A) = 30.7) while there are larger overlaps between empowered 
parliamentary debates and subsequent public space debates (LSq(B) = 10.7). A good 
example is the ‘women’s autonomy’ discourse: it is the most prevalent discourse in the 
first phase of public space debates (44.4 per cent), but then only constitutes roughly 
20 per cent of subsequent parliamentary and public debates. In Sequence 2, we can 
observe the reverse: empowered space debates more closely mirror preceding public 
space debates than in Sequence 1 (while the patterns of subsequent public space dis- 
courses deviate from empowered space patterns to a larger degree). Consequently, 
the transmission score that can be computed for Sequence 1 (TS = 45.93) is lower 
than for Sequence 2 (TS = 66.25). 


° Ina nutshell, this formula leads to low transmission scores in cases that display the following features: 
There is a low overlap between discursive patterns in the public space and in subsequent empowered space 
debates (high values for LSq(A)). This particularly points towards a domination of public space debates 
by empowered space debates if there is a pronounced overlap between these empowered space debates 
and subsequent public space debates (low values for LSq(B)) and if this overlap does not result from the 
fact that discursive patterns in public space debates before and after empowered space debates did match 
to a large extent in the first place (low values for LSq(C)). In Sequence 1 of the debate analysed here, 
we can observe that the discursive patterns of public debates preceding and succeeding parliamentary 
deliberations diverge significantly (LSq(C) = 26.1), meaning that the low value for LSq(B) actually indicates 
that public space discourses have been shaped by empowered debates. 
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Methodological tools that allow for applications of the basic approach summarized 
in Step 1 to Step 4 on a large scale present answers to one fundamental research- 
pragmatic challenge: Exploring and analysing the content (i.e. arguments or ‘topics’) 
exchanged between public and empowered spaces of a democratic system requires 
handling large amounts of textual data. Hence, it is recommended to utilize com- 
puterized content analysis methods. Along these lines, Beste uses an unsupervised 
computerized text analysis and statistical methods (iterative principal factor analy- 
sis) to uncover and model ‘topics’ or ‘frames in a set of textual data (2016, 303-304). 
Relatedly, Parkinson et al. (2020) apply Structural Topic Modelling (STM) to ‘the 
pool of perspectives’ and the spread of discourses throughout a variety of pub- 
lic and empowered spaces during the Scottish Independence Debate of 2007-2014 
and an Australian debate on constitutional issues (see also Bachtiger and Parkinson 
2019, 145). An assessment of transmissions at a large scale can therefore proceed by 
utilizing computerized content analysis tools in Step 1 to Step 3 shown below: 


Step 1: Content exploration—To explore the content of political communications 
at a large scale, researchers should apply an unsupervised automated 
content analysis tool, such as STM, which identifies ‘hidden semantic 
structures, patterns of co-occurring words in large amounts of textual 
data. 

Step 2: Identification of main discourses—Researchers must select and cluster the 
topics that they take to represent relevant political discourses. To make 
this selection in a methodologically guided and reliable manner, they may 
rely either on blended-reading procedures that supplement the comput- 
erized analysis with selected close readings (Lemke et al. 2015) or on 
complementary qualitative and secondary research (see Parkinson et al. 
2020). 

Step 3: Analysis of spread of discourses—After identifying the political discourses 
that occur in the overall corpus of textual data, researchers can analyse 
the discourses’ distribution across deliberative sites. Here, STM is a par- 
ticularly useful tool as it enables researchers to include document-level 
metadata referring to the time and locus of communication (Roberts et al. 
2014). Consequently, it provides a useful basis for studying ‘transmission- 
paths’—that is, to answer questions such as ‘where did the discourses 
appear first? or ‘What route did they “travel” through the political-societal 
system? 


In sum, my approach to measuring macro-deliberativeness combines established 
methodological tools to assess and then combine the measures for two core attributes: 
(a) deliberation in individual systemic sites and (b) transmissions. The application of 
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Figure 9.4 Discourses on Abortion Policies over Time 
Source: Author’s own graph. 


these tools to German abortion debates showed that they generally display high levels 
of meso-deliberativeness in both public and empowered spaces (see Figure 9.3).’° 

Yet, what is crucial for democratic systems’ macro-deliberativeness is not just that 
debates conform to procedural criteria such as respectfulness, reasonableness, etc. 
Rather, from a deliberative perspective, ‘democratic quality’ also requires the uptake 
of the contents of political communications, that is, of discourses or discursive ele- 
ments, in the empowered space. In other words: the ‘pool of perspectives’ of all 
affected citizens (Parkinson et al. 2020) needs to be represented in institutions that 
have the capacity to make collectively binding decisions. A mere accumulation of 
the scores representing the meso-deliberativeness in individual forums would not 
be sufficient to evaluate macro-deliberativeness. Researchers also need to study the 
transmissions between public and empowered spaces. While the democratic process 
assessed here achieved high meso-deliberativeness scores, it performed less well in 
this dimension: for Sequence 1 of the analysis period, the comparatively low transmis- 
sion score results from public space debates being strongly shaped by the discourses 
of empowered spaces, while public space debates had relatively low influence on 
the constellation of discourses in empowered space deliberations (see Figure 9.4). 
This highlights the merits of combining measures of debates’ meso-deliberativeness 
and of transmissions to arrive at a valid assessment of deliberativeness at a systemic 
level. 


1° Even compared to debates on other highly polarized issues—that are generally assumed to display 
lower levels of meso-deliberativeness than debates with low issue-polarization, the empowered space 
debates analysed are characterized by high levels of justification, respect, and constructiveness (see results 
from illustrative analysis as visualized in Figure 9.3 and Table 9.2; see also Bachtiger et al. 2008). 
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Measuring Deliberativeness at the Nation State Level: 
Concluding Recommendations 


To arrive at an aggregate measure of macro-deliberativeness, researchers need to 
combine the measures of meso-deliberativeness in deliberative sites assorted to the 
public and the empowered space of a democratic system with their measurement of 
transmissions. To ensure concept-measurement consistency, they should aggregate 
individual measures of meso-deliberativeness with the help of an additive procedure 
while the attributes (a) deliberation in individual sites and (b) transmissions should 
be combined with a multiplicative procedure (see section ‘Phase 3: Aggregation’)."* 

Throughout the previous section of this chapter, I illustrated applications of 
methodological tools by referring to German debates on abortion policies (1991- 
1993). While ‘abortion talk [may] tell [...] a great deal [...] about the nature and 
concerns of democracy as a whole’ (Ferree et al. 2002, 4), analysing a single policy 
debate certainly cannot suffice for assessing a democracy’s macro-deliberativeness 
in a given time frame. This points to a crucial research-pragmatic challenge: Even 
if researchers choose a comparatively short analysis period in which they wish to 
study a democracy’s macro-deliberativeness, they will not be able to study all for- 
mal and informal political communication in a country. To ensure the feasibility of 
their undertaking, researchers should therefore resort to theory-driven sampling of 
debates (see Bachtiger et al. 2008, 280-281). More specifically, researchers should 
analyse debates in public and empowered spaces that deal with both highly and 
less-polarized issues. Choosing exclusively debates on either highly polarized or 
very consensual issues would likely bias the assessment of democracies’ macro- 
deliberativeness (Bachtiger et al. 2008, 282). In selecting sample debates for assessing 
democracies’ deliberativeness, researchers should also account for pragmatic issues 
such as data availability: While there are comprehensive text corpora of parliamen- 
tary debates in many democratic nation-states,” and corpora of sample debates in 
traditional mass media are comparatively easy to assess and/or compile, analysing 
informal debates amongst citizens in online and offline spaces often poses more 
severe challenges and requires creating new corpora of textual data. 

Measurements of deliberativeness at a large scale involve ‘a classic empirical trade- 
off’ (Parkinson et al. 2020, 2) and require scholars to carefully balance real-world 
complexity and the demands of cross-system comparability and conceptual parsi- 
mony. Systemic deliberative theories provide adequate conceptual groundwork for a 
comparative assessment of deliberativeness at a large scale. This chapter demonstrates 


11 For example, for Sequence 1 of the abortion debate featured in the exemplary application of measure- 
ment tools above, this would amount to multiplying the arithmetic mean of Meso-Deliberativeness of the 
sample public space debates and empowered space debates with the Transmission Score (TS). 

12 For openly accessible text corpora of parliamentary debates, see, for example, ParlSpeech: (https:// 
dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/L4OAKN) or Clarin-D: (https:// 
www.clarin.eu/resource-families/parliamentary-corpora). 
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that fundamental methodological and pragmatic challenges associated with this 
undertaking can be overcome when we draw from established procedural guidelines 
of democracy measurements and utilize tools of electronic social science. 

Yet, this empirical trade-off also points towards the fact that assessing macro- 
deliberativeness with the toolkit outlined in this chapter is subject to the follow- 
ing caveats: First, the research-guiding question and the scale of analysis should 
indicate that more context-sensitive methods of inquiry are not feasible. Second, 
researchers should critically reflect on the procedures and results of building indices: 
Computations of ‘scores’ at a high level of aggregation are not a panacea (Bachtiger 
and Parkinson 2019, 150)—but they are a means for solving problems in compara- 
tive research. Meanwhile, macro-level assessments can serve as a basis for systematic 
comparative analyses that test and refine middle-range theorizations. They can pro- 
vide valuable insights into the effects of deliberative democracy and the conditions 
that facilitate and promote deliberativeness at a systemic level. 
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10 
Online Deliberative Matrix 


Raphaél Kies 


In the last decade, online deliberation has gained visibility in the fields of political sci- 
ence, political philosophy, political communication, and big data analysis. The term 
‘online deliberation’ corresponds not only to a topic of analysis that can be broadly 
defined as the quantitative and qualitative assessment of the political debates tak- 
ing place in the digital public sphere, but also to a model of e-democracy that aims 
to shape the different political usages of the web (Dahlberg 2011). In this chapter, 
I present one particular method to assess online deliberation: the Online Delibera- 
tive Matrix (ODM) (Janssen and Kies 2005). This matrix refers to the criteria aiming 
to contextually assess the quality of deliberation taking place in online spaces. It can 
be used for any political debate occurring on the web. 

The chapter is structured into three sections. In the first section, I argue that the 
ideal speech situation and the two-track model of democracy advocated by Habermas 
are still relevant theories to operationalize and assess online deliberation, especially 
from a systemic perspective. Next, I present a way of operationalizing online delib- 
eration by distinguishing between: (1) the criteria that can be used for assessing 
the visible presence of deliberation (inclusion, discursive equality, justification, reci- 
procity); (2) the criteria for assessing the deliberative attitude and intention of the 
active and passive users of the forum (reflexivity, empathy, sincerity); and (3) the cri- 
teria for assessing the expected outcomes of a deliberative process (plurality, external 
impact). In the final section, I briefly review several studies that fully or partially use 
the ODM to evaluate the deliberativeness of online debates in different contexts. 


Measuring Online Deliberation 


The ODM was first developed in 2005 (Janssen and Kies 2005) building on the first 
attempts to measure (online) deliberation (Schneider 1997; Dahlberg 2004; Graham 
2002; Steiner et al. 2005), and further elaborated some years later by reviewing the 
operationalization of certain criteria and how their ‘score’ should be contextually 
interpreted (Kies 2010). ODM takes the Habermasian idea of deliberative democ- 
racy as its normative starting point. As such, it refers to the ‘ideal speech theory’ 
(Habermas 1989) and the ‘two-track model of democracy’ (Habermas 1996). While 
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ideal speech theory provides us with the deliberative criteria to be measured (to 
be discussed below), the two-track model directs our attention to the levels of the 
opinion-formation and decision-making processes within which these deliberative 
ideals can be realized. The two-track model promotes a dynamic political process 
according to which the deliberative project is to be achieved as a result of the 
interactions of, on the one hand, the ‘strong publics’ (in particular, parliament and 
government) where decisions are taken, and, on the other hand, the weak publics (in 
particular, civil society) where new issues of public relevance can arise. Amidst these 
two publics is the general public dominated by the mass media (newspapers, televi- 
sion, radio) where the opinions and demands of the weak publics and those of the 
strong publics are presented and debated. In order to promote the appearance of a 
critical and informed public opinion, the media should be independent through the 
use of self-regulation procedures. They should also give visibility, without any distor- 
tion, to the opinions coming from the different publics, particularly the weak ones 
(civil society), which have lesser public and political influence (Habermas 2006). 

By referring explicitly to the systemic theory of Habermas, the ODM constitutes 
an early attempt to take into consideration the ‘systemic turm in the analysis of 
online deliberation (see Parkinson and Mansbridge 2012). The reference to the two- 
track model implies the necessity to adapt the deliberative criteria, in particular their 
evaluative standards, to the contexts in which the online forums are taking place. 
Accordingly, these standards should be defined by the external impact and the politi- 
cal/ideological beliefs of the social or individual actors hosting the online debate. The 
‘external impact’ is a proxy for distinguishing the weak-strong distinction present in 
the two-track model, which helps to identify the constraints and functions of a dis- 
cursive arena within the opinion- and decision-making process. In the next section, 
I will provide some examples of how the external impact influences the standard 
applied to several deliberative criteria. But before doing so, let me present the criteria 
considered to measure deliberation for any type of online discussion space. 

Table 10.1 lists nine deliberative criteria used in the assessment of online deliber- 
ation. As noted above, these criteria were developed through a close examination of 
the Habermasian ideal speech theory and the two-track model of democracy. In the 
next section, I will show how these criteria can be contextually operationalized. 


Operationalization of Deliberative Criteria for the 
Empirical Study of Online Sites 


For the empirical analysis, the criteria are merged in three analytical categories refer- 
ring to different aspects of the deliberation and methods of investigation. The criteria 
of inclusion, discursive equality, reciprocity, and justification seek to measure the 
apparent presence of deliberation, that is, the level of deliberation assessed through 
the analysis of the discursive architecture of the forum, and the dynamic and content 
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Table 10.1 Criteria for the Online Deliberative Matrix 


Deliberative 

criteria Meaning 

Inclusion All who are affected by and/or interested in the issues under 
discussion should be able to participate either actively or passively. 

Discursive Participants should have an equal opportunity to introduce 

equality and question any assertion whatsoever and to express attitudes, 
desires, and needs. 

Reciprocity Participants should listen and react to the comments formulated 
by other participants. 

Justification The opinions and propositions should be accompanied by 
accessible justification(s). 

Reflexivity Participants should critically examine their values, assumptions, 
and interests, as well as the larger social context. 

Empathy Participants should be sensitive to other views and opinions, not 
only of those present during the debates. 

Sincerity Participants must make a sincere effort to make known all relevant 
information and their true intentions, interests, needs, and desires. 

Plurality A deliberative context should be one in which a plurality of 
voices is heard even if these voices are critical of the dominant 
opinions/ideologies. 

External A successful deliberative process should have an impact on the 

impact opinions and decisions that are taken outside the context of the 
debate. 


of the discursive exchanges. Conversely, the criteria of reflexivity, empathy, and sin- 
cerity measure the non-visible aspects of deliberation. They focus on the conscious or 
even unconscious deliberative attitude and intention of the active and passive users of 
the online forums. Finally, the last two criteria, plurality and external impact, assess 
two expected outcomes from a successful online deliberative process: first, that it 
was able to confront a diversity of relevant voices and opinions; second, that it has 
an impact outside the online forum. To remain synthetic, I will not review in detail 
how other scholars have measured these criteria, but instead will focus on the ODM 
methodology.’ 


Criteria Related to Apparent Presence of Deliberation 


Inclusion 
The normative requirement of inclusion is that all those who are affected by, and/or 
interested in, the issues under discussion should be able to contribute and read 


1 For a review of the existing attempts to measure online deliberation see Kies (2010), Monnoyer and 
Wojcik (2012), Mendoza (2015), and Friess and Eilders (2015). — (here is the correct name Mendonca 
(2015)) 
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others’ contributions. The operationalization of the requirement of inclusion in an 
online environment implies that researchers assess mainly three types of threats for 
inclusion. 

The first threat is the digital divide, understood as the difference between people 
who have easy access to the Internet and those who do not. To test whether the dig- 
ital divide influences inclusion, we should observe, first, whether people have access 
to a device with an Internet connection and, second, whether they have the capac- 
ity to access and contribute to an online forum. Researchers can report descriptive 
statistics on Internet access, computer ownership, and survey data on Internet use 
and Information and Communications Technology skills. For example, several Euro- 
barometer surveys provide information to assess the connectivity and digital skills in 
EU Member States.” 

The second aspect concerns the moderation regime and/or technical architecture 
of an online forum as well as the requirements of registration and identification that 
can make access easier or harder and can thus be reported as factors that affect the 
level of inclusion. A moderation regime can vary a lot. It can be more or less intense: 
from no moderation at all to active moderation. It can occur before posting a mes- 
sage (i.e. pre-moderation), or after the posting of a message (i.e. post-moderation). 
Finally, moderation can be more or less conducive to inclusive and deliberative 
debate. The moderator can be a ‘censor —for example, by removing opinions that 
are at odds with the main ideology of the discussion space—or they can be a ‘pro- 
moter of deliberation’ by, for example, implementing a system of synthesis of debate, 
by giving more visibility to minority opinions, by offering background information 
related to the topics, etc. Some empirical findings suggest that well-balanced moder- 
ation is crucial to enable an inclusive and deliberative debate (e.g. Wright and Street 
2007; Janssen and Kies 2005; Edwards 2002). That being said, several studies also 
underscore that high levels of moderation can negatively impact the experience of 
deliberation on perceived procedural fairness, validity claims, and policy legitimacy 
(e.g. Meyer and Carey 2014; Perrault and Zhang 2019). As far as identification is 
concerned, it is important to be aware that while anonymity can increase the quan- 
tity of participation, it can simultaneously lower the quality of the content (Towne 
and Herbsleb 2012). 

Finally, researchers should also observe whether the institutions hosting and/or 
promoting the online forum were proactive in making the online forum as inclusive 
as possible. We should observe whether there were attempts to reach the cate- 
gories of people who would not spontaneously participate in online political forums 
(for example, older people, the less educated, those with visual impairments, etc.). 
Researchers are invited to observe, amongst others, whether the online forum was 
advertised in other media, whether specific training was planned for users, or if 
participation was encouraged through vouchers, money, gaming, etc. 


? See, for example, special EB (2020) ‘Attitudes towards the Impact of Digitalisation on Daily Lives; 
available at: https://ec.europa.eu/commfrontoffice/publicopinion/index.cfm. 
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Discursive Equality 

The normative requirement of discursive equality is that participants should have 
equal opportunity to introduce and question any assertion and to express attitudes, 
desires, and needs. We suggest that it should be measured by first assessing the ‘dis- 
tribution of voice’ in a conversation on the assumption that if only a small number of 
participants contribute in a large proportion, they dominate this debate. This idea of 
dominating the conversation is operationalized in a ‘participant contributions’ statis- 
tic. The distribution of the number of messages posted per participant may provide 
an indication that one group is dominating the conversation. This is not problematic, 
per se, as it is also a common experience offline that a minority of participants dom- 
inate a debate. However, it can be problematic if certain participants feel that their 
expression has been hindered by the dominant group. In certain cases, this can be 
deduced by reading the content of the forum itself (see empathy criterion), but ide- 
ally it should be measured through survey/interviews to find out whether participants 
feel that their views were suppressed (e.g. Graham and Wright 2014). In connection 
with the criterion of plurality, researchers are also invited to assess to what extent the 
active participants are representative of the internal and external variety of opinions 
(see plurality criterion). 


Reciprocity 

Reciprocity aims to assess whether participants listen and react to the comments for- 
mulated by other participants. If citizens do not listen to and interact with each other, 
there can be no deliberation, only monologue. The level of reciprocity should be mea- 
sured by assessing, at a basic level, the proportion of postings that are part of a thread 
versus those that initiate a thread (Schneider 1997). This measurement, however, only 
provides an approximation of the level of reciprocity. It is indeed possible that a mes- 
sage in the same thread is not a reaction to the content of a preceding message. A 
message in a thread can just as easily be a monologue without any reference to pre- 
ceding messages or the topic of the thread. For this reason, an in-depth evaluation of 
a message, which implies the reading of a message, is necessary in order to observe 
whether the message is a response to a past message contained in the forum. Here we 
suggest following the coding of authors like Jensen (2003), which uses the categories 
of ‘initiate’ (a message initiates a new debate), ‘reply’ (a message is a reply to a previ- 
ous message), and ‘monologue’ (a message is not really part of a debate). But even a 
more refined approach based on reading the messages can be problematic insofar as 
the absence of genuine reciprocity does not necessarily mean a lack of deliberation 
and, inversely, the fact that a message is reciprocal does not necessarily imply that 
this message is deliberative. 

This leads us to the question of interpreting the findings. What does it mean when 
the reciprocity level of a thread, forum, or news comment section is low? It can mean 
that participants do not listen to each other, but just state their opinions as well as 
frustrations: this is the typical dialogue of the deaf. The absence of reciprocity may 
also reveal a lack of interest in the topics discussed (Bentivegna 1998) or reflect a 
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general agreement about the topic discussed so that participants do not feel the need 
to react. Finally, a lack of reciprocity may also mean that (some) forum participants 
simply want to share information about a specific topic (see Kies 2010, chapter 6). A 
lack of reciprocity can have a multiplicity of meanings. To reach a refined evaluation 
of it, I suggest that no deliberative value should be given if it reflects an absence of 
interest, tacit agreement, sharing of information, etc. A negative deliberative value 
should only be given when the lack of reciprocity discloses an absence of a disposition 
to listen to each other. 


Justification 

The normative requirement for justification is that the opinions and propositions 
should be accompanied by accessible justifications. This means that there can be no 
deliberation if citizens appeal only to the authority of revelation or if their justifi- 
cation is based on complex arguments that only a restricted intellectual elite would 
understand (Gutmann and Thompson 2004, 4). With regard to the types of mes- 
sages for which the level of justification should be assessed, we argue, like Fuchs 
(2006), that it makes sense to measure the presence of justification only for messages 
containing ‘opinions’ and ‘suggestions. These are, after all, the messages for which 
a justification is expected. There is a plethora of attempts to measure justification 
that are more or less elaborated and relevant (see Kies 2010, 46-48). Our coding of 
justification is inspired by that of Jensen (2003). Accordingly, we should measure: 
(1) ifan opinion/suggestion is justified; and (2) if the type of justification is ‘internal’ 
(based on personal viewpoints and values) or ‘external’ (based on facts and figures). 
However, we reject the normativity of Jensen, in that arguments based on ‘objective’ 
information are better than those based on personal experience. As suggested by the 
two-track model of Habermas (1996) and by other authors (Mansbridge 1999; Young 
2000), justifications based on personal experiences can be as useful as those based 
on facts for the promotion of deliberative values. Particularly in weak and informal 
public spaces, the presence of internal arguments (storytelling, testimony, etc.) can 
enrich the debates and allow citizens who feel uncomfortable with rational-critical 
debates to express themselves freely. 

Researchers who are particularly interested in justification can also focus on 
the ‘level of justification. Here the coding schemes of Fuchs are also relevant. He 
measures ‘the complexity of arguments’ by distinguishing four degrees of argu- 
mentation: ‘one-dimensional message’ (no reason for holding an opinion); ‘simple 
message’ (one reason is communicated); ‘rather complex message’ (two reasons 
are communicated); ‘complex message’ (three or more reasons are communi- 
cated). The measurement of complexity of justification is optional. It should also 
be noted that its deliberative interpretation is particularly complex as an opin- 
ion does not necessarily need complex justifications in order to be sufficiently 
justified. 
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Criteria Related to the Deliberative Intentions 
of Participants 


Reflexivity 

Participants in a debate adopt a reflexive attitude if they critically examine their 
values, assumptions, and interests, as well as the larger social context. One of the 
defining features of reflexivity is ‘open-mindedness: This means that deliberating 
actors (e.g. citizens or public officials) must be ready and willing to change their 
opinions and preferences if they are sincerely persuaded that their initial opinions 
or preferences are incorrect or inappropriate for addressing the collective problems. 
Approaches based on content analysis attempt to find instances of reflexivity by read- 
ing the content of messages sent (Jensen 2003; Fuchs 2006), particularly whether 
these messages express opinion or preference change. Such a measurement is, how- 
ever, often meaningless insofar as it supposes that, in order to be deliberative, a debate 
should necessarily lead to persuasion, change of opinions or conflict resolution when 
in fact what the deliberative ideals require is the ‘readiness’ to change opinion if one is 
sincerely convinced by the arguments of someone else. An absence of opinion change 
should not be interpreted as non-deliberative behaviour if the concerned person is 
sincerely not convinced by another’s argument. Therefore, a different approach may 
be necessary to capture reflexivity as an internal process, and its manifestations may 
not always be expressed in an online forum. 

One possible approach to uncover reflexivity’s internal dynamic is to combine con- 
tent analysis with surveys and/or interviews. Surveys may also generate insight into 
the behaviour of ‘lurkers; who are by far the most numerous users of the forum. 
Lurkers can also be questioned about the way they have been influenced by the online 
debates. I suggest that the questions should focus on whether the active and passive 
participants of the forums ‘felt better informed’ and ‘took a moment to reflect about 
disagreements. There are several examples of how to formulate a questionnaire for 
the purposes of measuring reflexivity in different contexts (Jensen 2003; Hansard 
Society 2006; Fishkin 1995). 


Empathy 

The criterion of empathy can be defined as ‘the extent participants take into account 
and are sensitive to other participants and positions, not only those immediately present 
in the forum’ (Dahlberg 2004, 33). Alongside the criterion of sincerity, empathy is a 
cardinal indicator of deliberation, since all others derive more or less directly from 
it. Concretely, this means that if someone is genuinely concerned by the opinions 
and preoccupations of their fellow citizens, they will be more eager to interact with 
them (reciprocity), to justify their opinions (justification), and to change their mind 
if sincerely convinced by an argument (reflexivity). Since these criteria aim at the 
participants’ internal disposition—a proclivity for considering others’ opinions—its 
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measurement is complex and necessarily limited (e.g. Neumann et al. 2015). That 
being said, a partial measurement of empathy can be obtained by using several 
complementary methods: 


(1) Content analysis by counting cases of disrespect. These are cases such as 
personal attacks, dirty words, and xenophobic or homophobic declarations; 

(2) Survey/interview analysis by raising questions that seek to grasp whether par- 
ticipants entered into the debate with the intention of listening to others’ 
arguments and also to make other participants aware of external positions 
(that they do not necessarily share); 

(3) Deductively observing the scores of the deliberative criteria of reflexivity and 
sincerity, as they are generally positively correlated with empathy. If we find 
that the levels of reflexivity and sincerity are high in a forum, then we can 
suppose that participants are sensitive to the opinions of others and willing to 
reach a consensus. 


Sincerity 

Sincerity requires that all participants undertake a sincere effort to make known all 
relevant information and their true intentions, interests, needs, and desires. While 
sincerity appears to be a fundamental criterion, both for measuring deliberation for 
a specific case study and in a systemic perspective of deliberation (more on this to 
follow), it is to date the most poorly empirically investigated because it is also the 
most difficult to grasp. The presence of sincerity can nevertheless be grasped through 
different approaches. The first approach is to perform a qualitative analysis based on 
‘consistency in speech, consistency in speech and action, and coherence’ (Chambers 
1996, cited in Dahlberg 2002, 34). An inductive approach could look for instances 
where participants’ inconsistencies are exposed by other participants or it could seek 
out inconsistencies in speech and/or actions themselves. The second one is based 
on forum participants’ self-reporting (passive or active) using interviews and survey 
analysis. Since it is unlikely that participants who were not sincere would respond 
sincerely to an interview or survey, the question should be formulated in an indirect 
and subtle way. Lastly, as the perception of sincerity is also likely to have an impact 
on the dynamic and deliberative quality of debates, active and passive users should 
also be asked about whether they believe that other participants in the online forum 
were sincere. 


Criteria Related to the Deliberative Outcome of the Process 


Plurality 

The criterion of plurality aims to evaluate whether an online discussion hosts dif- 
ferent and divergent opinions. It is a fundamental criterion for evaluating how 
successful a debate has been in hosting and confronting all the relevant opinions ona 
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specific topic. It is linked to inclusion and discursive equality as both are determinant 
factors in promoting plural debates. I argue that the ideologies of the online delib- 
eration’s host or organizer are the best variable to define the standard applied to the 
criterion of plurality. For example, we could expect more diversity of opinions in an 
online forum hosted by a mainstream and neutral newspaper than an online forum 
hosted by a partisan newspaper. I would therefore put a higher standard of plurality 
for the first than for the second. 

A mixed-methods approach combining content analysis with survey should be 
privileged for assessing plurality. The content analysis of the web forum will allow 
us to assess the presence of different opinions on the issue at stake, while the survey 
analysis assesses the forum users’ diversity by focusing on socio-demographic pro- 
files (gender, age, education, occupation, etc.) and their political involvement and 
affiliation. The results of such surveys should, however, be carefully assessed since the 
socio-demographic homogeneity or diversity of a public does not necessarily imply 
that the debates will be homogeneous or plural (Best and Krueger 2005). In order 
to avoid such flaws, a method that combines content analysis with survey analysis is 
clearly recommended. 


External Impact 
A successful deliberative process should have an impact on broader public delib- 
erations taking place outside the online forum. I argue that external impact is the 
best proxy for the weak-strong distinction in the two-track model. Researchers must 
identify and define the constraints and functions of an online discursive arena for it 
to have impact in the opinion- and decision-making process in the macropolitical 
sphere. 

There are different methods through which to evaluate the external impact, 
depending on the type of forum analysed and the objective of the researcher. The 
external impact can be assessed: 


(1) By looking at whether there are concrete proposals of action. Jensen (2003) 
measures the external impact by assessing the presence of explicit signs that 
a sender is trying to extend discussions to an external agenda. He counts 
instances where participants attempt to ‘attract the attention of a politician, 
‘propose political actions; or ‘refer to external effect of a discussion’; 

(2) By assessing whether participants in an online forum are influential outside 
the forum (Hangemann 2002; Coleman et al. 2002; Beierle 2002). We suppose 
that the more well-known the participants in the forum are (e.g. journal- 
ists, politicians, intellectuals, influencers), the more likely they are to have an 
external impact; 

(3) By assessing whether decisions taken outside the online forum were inspired 
by opinions/proposals discussed in the online forum. This measurement is 
particularly common for discursive e-consultation processes. Such studies 
generally assess to what extent suggestions coming from the forum were 
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considered in the final drafts of policy proposals and whether participation 
in the forum led to more positive opinions of the institutions that hosted the 
forum (Beierle 2002; Coleman et al. 2002; Hansard Society 2006; Kies and 
Nanz 2013). 


The evaluative standards applied to the external impact should be adapted to the 
contexts in which the forum takes place. For example, the external impact require- 
ments should be more demanding for an institutional e-consultation process, than 
for an online forum without any institutional affiliations. This is because in a genuine 
institutional consultation process, the decision makers and/or the administration are 
bound to take into consideration the opinions and suggestions emerging from the 
online public. 


Illustrative Studies Referring to ODM 


There are several studies referring to the ODM to evaluate the quality of delibera- 
tion in different contexts. These can serve as concrete references for any researcher 
planning to use the ODM for their own research. For example, the ODM has been 
largely used to evaluate the level of deliberation of the Italian Partito Radicale (Kies 
2010, 117-142; Kies 2009), to compare the deliberative processes used in voting for 
proposals emerging from the online platform Plaza Podemos and the online develop- 
ment of the electoral programme of Barcelona En Comt (Bravo and Saez 2016), and 
to evaluate the most commented on citizens’ proposal discussed on the Barcelona 
government’ platform Decidim (Bravo et al. 2019). The ODM has also been largely 
used to compare two Slovenian online proposal portals: one emerging from the 
government and the other emerging from the citizen, with the objective of explor- 
ing whether the weak versus strong nature of the platform influences the quality of 
deliberation (Črnič and Prodnik 2015). 

Other interesting studies refer to the ODM to define and operationalize a limited 
number of deliberative criteria. This is the case with several experimental pieces of 
research that explore how certain factors affect the quality of deliberation, such as 
anonymity (Berg 2016), temporality (Strandberg and Berg 2015), and the effect of 
antisocial behaviours (Smith et al. 2012). The ODM is also a source of inspiration to 
evaluate the deliberativeness of interaction on social media. For example, it has been 
used to evaluate YouTube comments in respect of LGBTQ rights (Oliveira et al. 2014), 
the Facebook and YouTube channels of the White House (Halpern and Gibbs 2013), 
as well as user-generated political commentary on candidates’ Facebook pages during 
the 2008 and 2012 US Presidential elections (Camaj and Santana 2016). Finally, the 
ODM was used to review the deliberative quality of news website forums (Quinlan 
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et al. 2015), and of consultative debates on different topics and at different levels: from 
local (Sampaio et al. 2011) to European (Karlsson 2012; Kies and Wojcik 2011; Kies 
et al. 2013). 


Conclusion 


The objective of this chapter was to provide a synthetic overview of the ODM. It 
is important to underline the criteria it contains as well as their operationaliza- 
tion, remain open for discussion and improvement. Indeed, it is common to see 
research that utilizes the ODM, or other methods, as a starting point to develop 
their own methodology of investigation. This speaks to the fact that we are still far 
from having an uncontroversial method of investigation. In particular, the criteria 
aiming at assessing the deliberative attitudes of participants (ie. reflexivity, sincer- 
ity, empathy) could be improved by referring to the existing methods for assessing 
socio-psychological behaviours (e.g. Neumann et al. 2015). 

Potential improvements to the ODM may also emerge from the foreseeable shift 
from human coding to machine coding. The rapid growth in the usage of opinion 
mining and sentiment analysis (Piryani et al. 2017) is likely to evolve in the near 
future into instruments that can automatically measure the deliberativeness of a web 
forum. Such an evolution is to be welcomed, as it will allow for real-time measure- 
ment of deliberation in different contexts. But it also raises important questions. 
Outcomes may be misleading as automatic coding, even with the support of Artificial 
Intelligence, is unable to grasp certain linguistic nuances or expressions and is unable 
to provide contextual interpretations of the deliberative scores. More importantly, the 
vast amount of data that will be freely available through web-scraping could also be 
(ab)used for economic or political purposes, as is the case for the personal data that 
Internet users leave for free (Zuboff 2019). One should be aware of these dangers and 
elaborate guidelines and rules that can guarantee that online discussion forums are 
protected from such interferences. 

Another major challenge will be to adapt these criteria to the changing online dis- 
cursive environment. While until recently the usage of video-conference platforms 
tended to be limited to workplaces, since late 2019, in reaction to the COVID- 
19 pandemic, the usage of video-conferencing platforms (e.g. Zoom, Webex) has 
boomed in all sectors of life, leading to a hybridization of discursive exchanges— 
combining video, audio, and written exchanges. This major shift is not about to stop 
or reverse, and as such it will require new methods to evaluate the level of online 
deliberation, combining, for example, content analysis with body language and voice 
intonations. This will certainly enrich the possibilities for assessing online delibera- 
tion, but it is not likely to simplify the process. In any case, no matter how fast and how 
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profoundly the online discursive means and methodologies evolve, the ideal deliber- 
ative theory of Habermas and its followers remains a key reference point to preserve 
online public spaces from the current risks of colonization by economic and political 
interests. 
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11 
Experimental Methods 


Kimmo Gronlund and Kaisa Herne 


Early literature on deliberative democracy is largely normative. That is, it describes an 
ideal model of democracy. However, more recent research has also engaged in empir- 
ically testing the possibilities of achieving deliberation in actual political discourse 
and decision-making. One of the empirical methods in the deliberative democracy 
literature is experimentation. While experimentation in political science is by no 
means a novel invention (Gosnell 1926), it has seen a rapid rise over the last few 
decades (Morton and Williams 2010). At first sight, it might seem problematic to test 
something that is considered to present a normative model. However, experimen- 
tation on deliberative democracy aims at testing the consequences of institutions that 
include deliberation, and thereby providing a contribution to the design of democratic 
institutions. More broadly, like any normative theory, deliberative democracy has 
empirical elements, it makes certain claims about what would happen if ideal-type 
discussion were to take place, and such claims can be tested. 

Experiments in deliberative democracy have mainly concerned deliberative mini- 
publics, that is, more or less representative samples of the public deliberating on 
specific political issues. When we talk about deliberation or deliberative discussion, 
we refer to organized discussion that meets at least some minimum requirements, 
such as facilitated discussion that takes place in small groups, and that is guided 
by discursive rules that emphasize respect for others, listening to others, and jus- 
tification of one’s views (e.g. Gronlund et al. 2015). The main questions addressed in 
experiments have been whether and to what extent deliberative mini-publics achieve 
the kinds of outcomes that deliberative discussion is supposed to produce. Are opin- 
ions changed? Are they changed into a more tolerant direction? Is trust increased? 
Is the legitimacy and acceptance of decisions increased? In other words, would 
introducing the deliberative element into democratic institutions make democracies 
stronger? 

The chapter proceeds as follows. We first introduce the main characteristics of an 
experiment. Thereafter, we present the main object of experimental study in delibera- 
tive democracy, that is, deliberative mini-publics. That section takes up the strengths 
and challenges of engaging lay citizens in experiments, and specifically addresses 
data generation and analytical methods related to citizen deliberation. Finally, we 
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discuss the interpretations that can be made in relation to the theory of deliberative 
democracy based on experiments, and address some limitations of the experimental 
method. 


What Is an Experiment? 


The reason for using experiments is that they provide a control of variables not avail- 
able through naturally occurring data, and that this control makes it possible to detect 
causal mechanisms. For example, if we want to know how participation in a mini- 
public influences individual opinions, then organizing an experiment where some of 
the subjects participate in a mini-public, whereas others do not, allows for a compar- 
ison between these two groups. By this design, we are able to measure whether and 
how taking part in a mini-public influences opinions. Of course, real-world cases in 
which some people participate and others do not can also be observed. However, only 
in an experiment can we hold everything else between the two groups—those who 
deliberate and those who do not—constant, and thereby be confident that observed 
differences between the groups are due to experimental manipulation and not some 
other differences between the groups. 

An experiment is a controlled study of a causal relationship. By experimentation 
we can examine the influence of an independent variable on one or several dependent 
variables. In experiments, independent variables are those variables that are manipu- 
lated. They could also be called predictor variables. Dependent variables, in turn, are 
not manipulated. They are outcome variables whose values depend on the manipu- 
lated, that is, independent, variables. To take an example, a mini-public experiment, 
Setala et al. (2010) manipulated the small group decision-making method by using 
either a secret ballot or a consensual statement. In this study, the decision-making 
method was the independent variable and opinion formation the dependent vari- 
able. What distinguishes an experiment from other types of research that looks into 
causal relationships is the manipulation or treatment of an independent variable. An 
ideal experiment holds n-1 independent variables constant and manipulates only 
one. When other variables are held constant, we are able to say that any variation in 
the dependent variable is due to variation in the manipulated independent variable. 
Controlled variation is the key for detecting cause-effect relations. 

For example, if we are interested in the influence of deliberative discussion on opin- 
ion change, we give a part of our subjects the treatment, that is, have them discuss 
with each other in conditions that approximate those of ideal deliberative discussion. 
Another part of the subjects, the control group, does not discuss under ideal deliber- 
ative conditions. We then measure participants’ opinions pre and post taking part in 
deliberative discussion (the treatment group) or engaging in some other activity but 
not deliberating (the control group). 

In the case of deliberation, the control group can take several forms: it can sim- 
ply fill in the pre and post surveys with a certain time lag between the two surveys 
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(see Grönlund et al. 2015), or it can engage in free discussion without the elements 
of deliberative discussion (cf. Strandberg et al. 2019). The control group might also 
engage in some other activity, for example, writing down their ideas about the dis- 
cussed topic individually. Since the treatment group is compared to the control 
group, the point of comparison varies depending on what the control group does. 
For example, if the control group engages in free discussion and the treatment group 
engages in deliberative discussion, we can detect the difference between these two 
types of discussions—what is the added value of discussion being deliberative? On 
the other hand, if the control group writes down their thoughts individually while 
the treatment group engages in deliberative discussion, we can detect the difference 
between deliberative discussion and individual reflection. Sometimes experimental 
designs lack the possibility of a control group and different treatments serve as one 
another's controls. For example, in the aforementioned study on the influence of the 
decision-making method on participants’ opinion formation in a mini-public, the 
two treatment conditions were considered one another's controls (Setälä et al. 2010). 

In addition to using some kind of treatment, a random assignment to treatment 
and control groups is crucial to experiments. Randomization is needed because it 
rules out systematic variation between treatment and control group participants. 
Without randomization we cannot be sure whether observed differences are due to 
the experimental treatment or to systematic variation between the participants in the 
treatment and control groups. It is noteworthy that participants in face-to-face mini- 
public experiments have to show up at a certain place and invest their time, as well 
as other resources, in deliberative discussion. It is obvious that this investment can 
influence the type of people who are likely to attend. For this reason, we cannot rule 
out systematic variation if randomization to treatment and control groups is done 
amongst all those invited to take part. A better strategy is to randomize amongst those 
who are willing to participate, or better still, if possible, amongst those who show up. 
The reason is that those who are invited, those who are willing to participate, and 
those who show up might represent slightly different types of groups. This means 
that we cannot be sure whether the observed outcome variation occurs because of 
our treatment or rather because of the differences between the types of people (not) 
willing to participate. 

In addition to the control of variables, a clear advantage of experiments is that they 
can be replicated. Replication of research is not easily available through naturally 
occurring data. However, replication is crucial because it increases the reliabil- 
ity of observed results. For example, if we obtain similar results from mini-public 
experiments organized in different countries, we can be more confident about the 
generalizability of the results. 

In social sciences, three different types of experiments can be separated. These 
types can roughly be characterized according to a scale from more control to less 
control, and from less external validity to more external validity. Laboratory exper- 
iments allow for most control but are often criticized for a lack of external validity. 
The critique usually stems from the use of student subjects and conditions that lack 
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similarity with real-world circumstances. For this reason, there has been a call for 
more realistic field experiments conducted in participants’ natural environments. 
While field experiments bring about certain advantages, the drawback is that they 
cannot achieve the same degree of control as laboratory experiments. Lab-in-the- 
field experiments fall between pure field and pure laboratory experiments and are 
based on an idea of increasing external validity, for example by bringing experiments 
into natural locations without losing much of the control of variables. Mini-public 
experiments based on a random sample of the population and taking place either 
online or face to face commonly fall into the category of lab-in-the-field experi- 
ments. However, drawing the line between different types of experiments is not 
unambiguous. 

While randomization is commonly seen as essential to experimentation, there are 
experiments that lack random assignment. In quasi experiments, subjects are not ran- 
domized into treatment and control groups. There are two types of quasi experiments, 
depending on whether the experimenter has control over the assignment of subjects 
into treatment and control conditions. In naturally occurring experiments, the exper- 
imenter has no control over the assignment of experimental subjects into treatment 
and control groups, but these groups are formed in a natural process. An example is 
a study which compares Brazilian municipalities that use or do not use participatory 
budgeting (Touchton and Wampler 2014). Naturally occurring experiments have the 
lowest level of control over variables, but they provide real-world cases that resemble 
experiments. When the experimenter has control over the assignment in a quasi- 
experimental design, subjects are allocated into treatment and control conditions 
based on a certain characteristic. 


Experiments and Deliberative Democracy 


Experimental research is usually motivated by a willingness to test theories. Based 
on the theory or theories, specific hypotheses are formulated. These are then tested 
with an appropriate experimental design. Theories are challenged or given support 
depending on the results of the experiment. However, in the case of deliberative 
democracy, the relationship between theory and empirical research is rather atypical 
because deliberative theory is a normative characterization of ideal states of affairs. 
While normative theories cannot be tested as such, they involve empirical claims that 
can be tested, and this is what experiments on deliberative democracy seek to achieve 
(Setälä and Herne 2014). 

It is also notable that experiments are closely related to the central concepts of 
deliberative democratic theorizing. Experiments in deliberative democracy study the 
consequences of taking part in deliberative discourse. They often study how opin- 
ions change during deliberation, for example whether they become more similar 
or whether a meta-level consensus is achieved (Dryzek and Niemeyer 2006). Many 
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experiments also measure different civic virtues central to deliberative democracy, 
for example social and political trust, political efficacy, political knowledge, and 
readiness to participate in politics. Moreover, many experiments touch upon the 
legitimacy of decisions and procedures by asking participants how they conceive the 
decision-making processes and decisions made in them. 

The usual motivation for experimentation is to test a given theoretical model. 
However, in the case of deliberative democracy, the motivation for experimentation 
can also be a study of discursive democratic innovations not yet used in real-world 
democratic systems. Via experiments, we can study the potential consequences of 
certain ways of organizing democratic participation, and, on the basis of the results, 
decide whether these ways should be put to use to improve the quality of democracy. 
Experiments can thereby contribute to the design of democratic institutions. 

The focus of this chapter is on experiments with deliberative mini-publics. The 
reason for this is simple: experimental research on deliberative democracy has so 
far mainly concerned mini-publics. The reason for the popularity of mini-public 
experiments is likely their usefulness to test issues, such as opinion change, discus- 
sion dynamics, as well as various ‘side effects’ of deliberation, for example, political 
and social trust, political knowledge, and political efficacy (Grönlund et al. 2010; 
Gronlund 2016). 

One factor that varies between different mini-publics is size, that is, the number 
of participants. Even though all mini-publics have the goal of being representative 
in the sense of representing different viewpoints amongst the public, most mini- 
publics fall short of being representative in a statistical sense and none of them are 
representative in the electoral sense (Goodin and Dryzek 2006). The best-known 
models for organizing mini-publics are Citizens’ Juries (12-26 participants), Consen- 
sus Conferences (10-50 participants), Citizen Assemblies (50-160 participants), and 
Deliberative Polling (100-500 participants) (Breckon et al. 2019). Of these models, 
the most widely implemented mini-public is the Deliberative Poll”, developed by 
James Fishkin (1991). Deliberative Polls have by now been organized over 100 times 
in twenty-eight countries (Fishkin 2018). The Deliberative Poll model has become so 
widely known that Jane Mansbridge (2010) has dubbed it the ‘gold standard’ of how 
to organize deliberative mini-publics. 

In many cases, Deliberative Polls and other mini-publics involve policymakers 
(Grönlund et al. 2014, 1). Involving external actors, such as national, regional, or 
local governments, is good if your aim is to make deliberative democracy ‘bite; that 
is, to connect it with the democratic system. Yet, from a scientific viewpoint, it is likely 
to decrease the possibilities to design experimental manipulations freely. Since the 
scientific aim of most deliberative mini-publics is to study how the participants are 
affected by deliberation, this may decrease the internal validity of the mini-public as 
an experiment. Thus, for a researcher there is a trade-off between choosing to stim- 
ulate public debate (or even to have a real impact on democratic decision-making) 
on the one hand, and designing a coherent controlled experiment on the other. This 
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is something that the researcher should decide early in the planning process. By the 
time external actors are invited, it might be too late. Sometimes external actors are 
necessary for the simple reason that the researcher needs funding for her experiment. 

In mini-publics, deliberation normally takes place in small groups; it is guided by 
discussion rules, and facilitated by a trained moderator (e.g. Gronlund et al. 2015). 
The rules emphasize inclusiveness, equality, and respect for others, as well as listen- 
ing and being open to others’ views. The recruitment of participants can vary. For 
experimental purposes, a convenience sample (of students) can be appropriate, but in 
order to increase external validity, a representative sample of the population is better. 
Sometimes open calls through media and advertising campaigns are used, but this, 
of course, might create a bias amongst participants because of self-selection. Before 
discussing, participants can be given balanced information about the topic, and they 
may hear and pose questions to an expert panel (expert panels are always part of the 
Deliberative Poll). Participants usually fill in a survey before and after taking part in 
deliberation. These pre and post surveys are then used to study potential changes in 
different outcome measurements, such as opinions. Small group discussions are also 
commonly recorded and transcribed and can thereby also be analysed. 


Example: An Experimental Study of Enclave Deliberation 
and Group Polarization 


In this section, we describe what a controlled experiment in citizen deliberation might 
look like. In other words, we look at an experiment that was fully funded and designed 
by academics and did not have a direct connection to a political process. We orga- 
nized this experiment in 2012 (Grönlund et al. 2015). We were able to cover the costs 
through project funding from the Academy of Finland (project no. 251222), and 
could therefore organize it as a controlled lab-in-the-field experiment. In designing 
the experiment, we were inspired by the concern, most notably expressed by Cass 
Sunstein (2002, 2007, 2009), about the increased tendency to only discuss politics 
in like-minded groups. This ‘enclave deliberation may lead to group thinking with 
extreme views and an amplification of cognitive errors as a result. When like-minded 
people discuss among themselves, they reinforce each other’s views which prevail in 
the group at the outset. After discussion, their views move in the direction of the ini- 
tial bias. This phenomenon is called group polarization. Sunstein (2009, 3) defines 
group polarization as follows: ‘members of a deliberating group usually end up at 
a more extreme position in the same general direction as their inclinations before 
deliberation began. Like-minded discussion may also lead to an amplification of cog- 
nitive errors (Sunstein 2007, 80-95, 140-143), meaning that people's false beliefs are 
strengthened. When it comes to deliberation, the presence of conflicting viewpoints 
is often regarded as a necessary condition (Thompson 2008, 502). However, the term 
‘enclave deliberation’ has been used to refer to any discussion amongst like-minded 
people (Sunstein 2002, 2007, 2009). We wanted to test how deliberative norms work 
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in like-minded groups. Our central research question was: ‘Can group polarization 
be avoided?’ 

The topic for the deliberative experiment was immigration. The participants’ 
opinions on immigration were measured before and after deliberation with a ques- 
tionnaire. Based on their baseline views, respondents were divided into two enclaves. 
Those with negative attitudes to immigration formed a con enclave, and those with 
a positive view on immigration formed a pro enclave. Since the main research inter- 
est was enclave deliberation, we manipulated the group composition in order to 
compare deliberation in two types of groups: (1) groups with similar views on immi- 
gration; and (2) groups with diverse opinions on immigration. Thus, the participants 
were randomly assigned to like-minded groups, mixed groups, and a control group. 
The treatment groups deliberated, whereas the (pseudo) control group only filled in 
surveys at home. 

First, we mailed out a short survey to a simple random sample of 12,000 adults in 
the region of Turku (Abo). The response rate to this survey was 25 per cent. It con- 
sisted of fourteen items measuring attitudes on immigration. Since the design of the 
experiment required people with clear views on the immigration issue, we excluded 
moderates (n = 631), that is, those respondents whose opinions on immigration were 
close to the median value of the frequency distribution (see Grönlund et al. 2015 for a 
detailed description). Then, a second survey (T2), which also included an invitation 
to take part in a deliberation event, was sent out to 2601 people. Half of the invited 
sample consisted of pro- and the other half of anti-immigrants (con). We also clarified 
that only a part of those who volunteered could be included in the deliberation event 
and that the choice would be made by lot. Each participant in deliberation received 
a remuneration of 90 euros, whereas the control group received 15 euros. 

Ultimately, 805 people volunteered, and we invited 366 people to take part in the 
deliberative experiment. The target sample was 256 participants, that is, thirty-two 
small groups of eight participants each. Alas, only 207 people showed up. Especially 
people in the con enclave tended to abstain at this final stage, even though there 
were no indications of this kind of a bias earlier. For the experiment, we randomly 
assigned people into treatment conditions and small groups within the con and pro 
enclaves. In the end, we were able to form ten pro like-minded groups, five con like- 
minded groups, and eleven mixed groups. Because of attrition at the last stage, and 
the need for balance between the enclaves, each mixed group consisted of exactly 
eight participants—four from both enclaves—whereas the group size was allowed 
to vary between seven and nine in the like-minded treatment. The pseudo control 
group consisted of 369 people. The deliberation event took place during the course of 
a weekend in the spring of 2012. Each participant took part during one day. The setup 
was identical. Both days started with a fifteen-item knowledge quiz, after which the 
participants were briefed in plenum about some basic and balanced facts related to 
immigration in Finland. The briefing was also given as a handout to the participants. 

Deliberation took place in small groups and lasted for four hours, including a forty- 
five-minute lunch break. Trained moderators facilitated the discussions and made 
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sure that the discussion rules were followed. First, each participant put forward a 
theme related to the immigration issue. The discussion started with these issues and 
was free—the moderators interfered only if any of the group members dominated or 
completely withdrew from the discussion. After deliberation, each participant filled 
in a post-test survey. 

The main result of the experiment was that all participants in the anti-immigration 
enclave became more liberal. This was especially true in the mixed treatment, 
where the groups consisted of four anti- and pro-immigrants each, but also in the 
like-minded groups consisting of participants with initially negative views on immi- 
gration. Within the pro-immigration enclave, participants in the mixed treatment 
did not change their preferences, whereas a slight polarization into a liberal direc- 
tion could be traced in the pro like-minded groups. Especially those pro-immigration 
participants in like-minded groups who did not learn in the course of deliberation 
became polarized, that is, even more liberal in the post-deliberation measurement 
(Grönlund et al. 2015). The results support a central theoretical assumption in delib- 
erative democracy, which claims that all opinions should not have an equal weight 
in the process of public reasoning. Reasonable arguments appealing to generalizable 
moral principles should be powerful, whereas arguments based on attitudes such as 
prejudice should be ‘laundered’ in the course of deliberation (Goodin 1986). Thus, 
our interpretation of the outcome of the experiment was that deliberation is different 
from free discussion. The deliberative package with information material and discus- 
sion rules emphasizing respect, equality, and reflection can be particularly useful if 
we want to prevent group polarization in like-minded groups (Grönlund et al. 2015). 

The data that can be collected through a deliberative mini-public experiment are 
plentiful. First, survey instruments are good for tracing changes in opinions. Com- 
parisons can be made in two ways: (1) within treatments, using the pre-test, post-test 
design and measuring the same individuals before and after deliberation; (2) a post- 
test-only strategy can be used by comparing treatment group(s) with a control group. 
Both methods have some weaknesses. Using the within treatments method, a phe- 
nomenon called ‘regression to the mean’ might be a problem. Regression to the mean 
is a phenomenon where measuring the same sample twice—such as in a pre-test, 
post-test design—tends to lead to regression towards the group’s natural mean upon 
the second measurement, creating an illusion of a treatment effect (Torgerson and 
Torgerson 2008, 10-15). Using the between treatments method, one must be certain 
that the randomization has succeeded, and that the treatment and control groups are 
not different in certain characteristics, such as education, age, or gender. In small 
samples, this might be an issue. Thus, if possible, a combination of within treatments 
and between treatments testing can reduce the risk of false findings, especially to 
reduce type I errors caused by regression to the mean. 

Second, data can be generated to analyse the contents and dynamics of delibera- 
tion. For this, the discussions need to be at least audio recorded and transcribed. Such 
endeavours are often time-consuming, at least with rarer languages such as Finnish 
and Swedish, involving manual transcriptions and coding (Himmelroos 2012; 
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Himmelroos and Christensen 2014), but they can also be automated, as Kaiping Chen 
(2019, chapter 2) demonstrates. There is no clear-cut standard for content analysis 
of deliberation, but the most promising tool for measuring the ‘deliberativeness’ of 
discussions is the Discourse Quality Index (DQI), developed by Steenbergen et al. 
(2003), through an analysis of discussions in the British House of Commons. Another 
way of analysing deliberation in small groups is to look at the dynamics with the help 
of network analysis (Gerber 2014): who addresses whom and how? For analysing the 
dynamics of the behaviour in groups, video recording is another possibility, but the 
interference of cameras would, of course, present another treatment, which ideally 
should be controlled for (microphones are less conspicuous than cameras). In addi- 
tion, the moderators can be surveyed or interviewed afterwards. We have used this 
method in order to trace social pressures at the small group level, and compared these 
surveys with similar survey questions to participants (Setälä et al. 2010). 


Conclusion 


Experiments on deliberative democracy most commonly study micro-level phenom- 
ena. A typical deliberative experiment asks what kinds of individual-level outcomes 
certain institutional arrangements have. For example, we can test whether deliber- 
ative discussion produces the types of outcomes it is claimed to produce, or study 
more specifically different ways to organize deliberative discussions and study their 
consequences. Apart from naturally occurring ones, experiments are not well suited 
to studying macro-level phenomena because control and experimental manipulation 
are hard to achieve at the macro level. However, naturally occurring experiments 
allow for a study of macro-level effects. An example is the use of democratic innova- 
tions in certain municipalities and not in others (e.g. Touchton and Wampler 2014). 
Comparing those municipalities, where democratic innovations are used to those 
where they are not, can also be done with regard to certain macro-level indicators, 
for example, voter turnout. In principle, it is also possible to study cross-country 
variation, but differences between democratic systems makes this even harder than 
comparing variation within a country at the local level. 

Controlled experiments are harder to implement at the macro or even meso level. 
Therefore, experimental research is not likely to give answers to current issues on the 
systemic nature of deliberative democracy (Parkinson and Mansbridge 2012). We 
are not able to experimentally manipulate the deliberativeness of a whole democratic 
system and test the consequences. 

Although a large number of experiments with deliberative mini-publics have 
already been carried out, it is clear that there is room for more experimentation in the 
field. We separate four areas of interest where we see a need for further experiments, 
but it is likely that there are other relevant directions for future research. Controlled 
experiments are typically conducted within a certain context, whereas systematic 
comparative experimental research, for example, between countries, is lacking. Thus, 
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it would be important to obtain knowledge of the context dependency of experi- 
mental results, and to increase the external validity of the results. Second, there are 
a number of ways to organize mini-publics, and we lack comparative evidence on 
the influence of different types of mini-publics on the consequences of deliberation. 
Third, we also lack knowledge as to how certain individual-level psychological pro- 
cesses interact with participation in deliberative discussion (Mercier and Landemore 
2012). We know that people tend to be biased in their information processing, that is, 
mainly seeking evidence that confirms their existing views (Lodge and Taber 2013), 
but we do not know whether taking part in deliberative discussion would allevi- 
ate these types of biases. And finally, while it seems to create certain problems, it 
also seems essential to run mini-public experiments that have connections to real- 
world political decision-making in order to be able to test how being part of a real 
decision-making process influences the consequences of deliberation, both amongst 
the participants and in the mini-public as a whole. 
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12 
Deliberative Field Experiments 


Jon Kingzette and Michael Neblo 


In recent years, scholars and practitioners of deliberation alike have begun con- 
ducting ‘field’ experiments. By field experiments, we mean to designate systematic 
attempts to understand the causal dynamics of deliberation by manipulating features 
of the system: (1) in naturalistic settings (rather than merely via surveys or in labs); 
(2) on real political issues (rather than merely hypothetical scenarios); and (3) engag- 
ing a broad cross-section of people, generally in a specific political jurisdiction 
potentially affected by pending political action (rather than merely whichever col- 
lege sophomores are at hand). We typically conduct field experiments—as opposed 
to other types of experiments (on scenario experiments, see Werner and Muradova, 
Chapter 13 in this volume; on experimental methods, see Gronlund and Herne, 
Chapter 11 in this volume)—to enhance the scientific generalizability and practical 
relevance of the research. That is to say, scientifically, well-executed field experiments 
enhance the external validity of the results, and practically, they can contribute to real 
politics (Neblo et al. 2017; Green et al. 2019; Esterling et al. 2022). 

Below, we discuss the concept of a deliberative field experiment, how it relates to 
broader theories of deliberative democracy, how and when researchers can use delib- 
erative research experiments, what types of research questions they help to answer, 
and how to manage trade-offs between internal and external validity. We also con- 
sider challenges associated with conducting deliberative field experiments including: 
(i) differential attrition; (ii) practical and ethical issues related to conducting research 
‘in the wild’ of real politics; (iii) difficulty in manipulating (or even determining) the 
‘dosage’ of the treatment; and (iv) difficulty in distinguishing the precise mechanisms 
behind effects. 


AConception of Deliberative Field Experiments 


As with all experiments, assignment into treatment is foundational for all inferences 
in field experiments. However, unlike laboratory or survey experiments, field experi- 
ments attempt to ‘simulate as closely as possible the conditions under which a causal 
process occurs’ to maximize generalizability (Gerber and Green 2011, 2). In the case 
of deliberative field experiments, the causal process that researchers aim to capture 
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is how people make collective recommendations (and occasionally decisions) on politi- 
cal issues when they are given an opportunity to deliberate the issues with each other. 
This conception of deliberative field experiments reveals four ways to delimit its 
boundaries vis-a-vis other types of experiments and non-experimental research on 
deliberation. 

First, deliberative events must involve assignment into treatment and analyses that 
examine the impact of treatment to count as deliberative field experiments. This is 
important to emphasize because many well-known ‘experiments in deliberation, used 
in the colloquial sense, are not experiments in the analytical sense used here. For 
example, there has been excellent research on the Oregon Citizens’ Initiative Review 
(e.g. Knobloch et al. 2013), but the main process employed by the Review does not 
involve constructing a control group nor does it involve measuring pre-discussion 
attitudes necessary for a repeated measures experiment. In short, it is very much in 
the field but not an experiment in the strict scientific sense. 

Second, by how people make collective recommendations, we mean that when peo- 
ple participate in a deliberative event, they must expect the output to be (at least) 
considered by decision makers for it to be a deliberative field experiment. This 
separates deliberative field experiments from studies of what Mansbridge (1999) 
labels ‘everyday political talk (e.g. Cramer Walsh 2004; Mutz 2006), because out- 
puts of ordinary political conversations are not expected to be considered by decision 
makers. To be clear, we believe that educational forums and everyday political talk 
are important elements of a civic culture. They are just not the object of study in 
deliberative field experiments. 

Third, by using the phrase on political issues, we mean that discussions must be 
on a real, pending political issue for them to constitute a deliberative field experi- 
ment. While we do not have the space here to dwell on what exactly we mean by 
‘political’, one barometer for assessing whether a discussion covers a real-world issue 
is by determining whether some ‘public’ not participating in the experiment could 
potentially have some knowledge of the issue. For example, in a study investigating 
moderator effects, Spada and Vreeland (2013) conducted a deliberative experiment 
in a college course in which students deliberated on the slate of course requirements 
for the class. On our conception, this is not a deliberative field experiment because 
no one outside the class would have any knowledge of the issue. 

Fourth, a deliberative experiment must occur in a naturalistic setting for it to be 
a field experiment. This easily disqualifies laboratory and survey experiments, but it 
does leave some ambiguity about what counts as ‘naturalistic. Indeed, this leads to an 
important aspect of deliberative field experiments which makes them distinct from 
most other field experiments. Most scholars who conduct deliberative field exper- 
iments are typically interested in seeing what happens when people participate in 
deliberative events that are explicitly designed to elicit good outcomes (Goold et al. 
2012). Akin to a drug trial, the hope is that deliberative field experiments create a 
healthier civic culture compared to the status quo. To achieve this goal, scholars 
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use a variety of tools, such as inviting expert panellists to provide information to 
participants (Fishkin 2018), providing written educational resources (Carman et al. 
2015), or inviting a representative sample of constituents to a town hall with leg- 
islators (Minozzi et al. 2015). In the next section, we review theoretical reasons 
for this, but there is also a more fundamental ethical consideration. Because out- 
puts from deliberative field experiments aim to inform decision-making on political 
issues, there is an ethical obligation to strive for high-quality deliberation, even while 
recognizing that there will always be shortcomings. 

This is important because it changes how external validity ought to be assessed. 
Whether a process used in an experiment mimics aspects of political conversations 
as they typically occur is beside the point. Instead, the external validity of deliberative 
field experiments should be assessed based upon the extent to which the experimen- 
tal environment is suited for answering a specific research question. For example, 
if a research question asks how deliberating in a group of co-partisans on a highly 
polarized policy issue influences affective polarization, then a study ought to ensure 
that co-partisans are indeed discussing a highly polarized issue with one another (in 
the treatment group). Moreover, because results may be contingent on the specific 
issue being discussed, researchers studying this question would enhance the general- 
izability of the study if they randomly assigned participants to discuss one of several 
highly polarized issues, instead of only one. Researchers would also enhance external 
validity if they studied this question amongst members of multiple political parties 
instead of one, etc. 

While this example is necessarily limited, our hope is that it shows that achiev- 
ing high external validity in deliberative field experiments is still challenging, even 
using these standards. Additionally, we urge caution in interpreting the results of 
a single deliberative field experiment. A study showing that Republicans’ levels of 
affective polarization increase after deliberating about immigration policy does not 
definitely demonstrate how discussing any highly polarized issue influences affec- 
tive polarization. In short, deliberative field experiments aim to provide a space to 
observe politics as it could be with innovations or reforms, not politics as it is under 
status quo conditions. 


Deliberative Field Experiments and Conceptions 
of Deliberative Democracy 


Field experiments are well suited for studying how attitudes and behaviours are 
impacted when people deliberate in spaces curated to foster positive outcomes. 
This makes them excellent tools for studying conceptions of deliberative democ- 
racy (Bachtiger et al. 2010; Neblo 2007), which often fuse descriptive theories about 
how politics actually works and normative theories about how politics ought to work 
(Chambers 2003; Gutmann and Thompson 1998; Habermas 1996; Neblo 2015). 
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For example, a central feature of Gutmann and Thompson’s (1998) normative the- 
ory is that for a conversation about public policy to be truly deliberative, all parties to 
the conversation need to display mutual respect, listen to each other, and refrain from 
using arguments based upon self-interest. Additionally, these conversations should 
be inclusive of all stakeholders. Deliberative field experiments are capable of testing 
how incorporating these features into a deliberative environment influences citizens’ 
attitudes. Price and Cappella (2002) did just this, showing how online deliberations 
can increase social trust and political engagement. 

Similarly, Neblo, Esterling, and Lazer (2018) develop a variant on standard 
accounts of deliberative democracy tailored to the representational needs of modern, 
mass polities. The concept of ‘directly representative democracy’ hinges on the notion 
that elected representatives and their constituents should interact with each other on 
the issues in two ways. The first is ongoing republican consultation, which involves 
representatives seeking input from their constituents on public policies. The second is 
ongoing deliberative accountability, which includes representatives providing expla- 
nations for their policymaking activity to constituents. After laying the theoretical 
groundwork, their experiments then generate data on how citizens’ attitudes change 
when representatives engage in these activities through online Deliberative Town 
Halls. 

These are just two examples of how deliberative field experiments respond to 
theories of deliberative democracy. However, they are used to study many more phe- 
nomena (Kielty et al. 2021; Minozzi et al. 2022). In the next section, we review a 
larger swathe of research questions that deliberative field experiments have aimed to 
address. 


What Kinds of Questions Can Deliberative Field 
Experiments Answer? 


In deliberative field experiments, scholars manipulate several aspects of the delib- 
erative environment (Gastil 2018, 277). These manipulations often include varying 
‘inputs’ —for example, whether and how participants receive information about the 
issue (Carman et al. 2015; Neblo et al. 2018). Researchers also manipulate attributes 
of the deliberative process itself such as the composition of discussion groups. For 
example, Caluwaerts and Reuchamps (2014) randomize whether participants are 
placed in homogeneous or heterogeneous discussion groups based on language, 
Esterling et al. (2015) use a natural experiment in which placement into discussion 
groups was random to study how heterogeneity of opinion influences satisfaction 
with the forum, and Humphreys et al. (2006) examine a deliberative event in which 
discussion leaders were randomly assigned. These articles respond to findings that 
homogeneous discussion groups tend to increase polarization (Sunstein 2002) as well 
as concerns about people with more power or knowledge (Sanders 1997) causing 
opinions to change in their preferred direction during deliberation. 
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While the above examples all involve manipulating specific aspects of the delib- 
erative environment, some scholars study the total effects of attending a deliberative 
event. For example, Luskin, Fishkin, and Jowell (2002, 459-460) describe delibera- 
tive polling as a ‘grand treatment’ that involves participants anticipating the event, 
reading an issue brief with information about the issue they will discuss, engaging 
in small group discussions with other participants, hearing experts and politicians 
answer questions about the issue, and socializing with other participants outside the 
confines of official study activities (since deliberative polling typically involves bring- 
ing participants together from all over a country to one hotel). The goal of this process 
is to observe public opinion on an issue when people are informed about it and have 
an opportunity to deliberate on it with fellow citizens. As Fishkin (2018) reviews, 
deliberative polling has now occurred all over the world, and has also taken place in 
different types of political jurisdictions, but in each case the conveners have taken 
pains to create ‘on ramps’ into real political decision-making. 

In short, scholars have conducted deliberative field experiments to study particu- 
lar parts of deliberation and to examine the total effects of bringing ordinary people 
together to deliberate on real policy matters. We now turn to discussing the pro- 
cess of conducting a deliberative field experiment and the challenges of doing so. 
Our hope is that these sections function as a practical guide for scholars consider- 
ing deliberative field experiments in addition to providing a brief overview of the 
process. 


The Process of Deliberative Field Experiments 


The process of conducting deliberative field experiments varies widely depending 
on whether deliberations are online (e.g. Neblo et al. 2018; Nyerges and Aguirre 
2011; Price and Cappella 2002), face to face (e.g. Luskin et al. 2002), or both (e.g. 
Carman et al. 2015; Luskin et al. 2004; Vlahos et al. 2021), how informational mate- 
rials are provided to participants, whether experts are involved in the deliberative 
process, and how many different discussion groups are created. However, as noted 
previously, features shared by all deliberative field experiments include the results 
of the forum somehow being available to relevant decision makers, the discussion 
being focused on a real political issue, and the experiment designed explicitly to 
achieve good outcomes. Additionally, though the particulars are different for each 
field experiment, we can speak in general terms about the process required for 
them. 

First, researchers conducting deliberative field experiments need to recruit partici- 
pants into their study (Neblo et al. 2010). The population from which participants are 
drawn depends on what decisions are being informed by the results of the forum and 
where those decisions are being made. Most existing research has involved studying 
people in a well-defined political constituency, which makes participants constitute 
a mini-public. As Warren and Gastil (2015, 567) write of mini-publics: 
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These bodies are comprised of anywhere from 20 to 200 or more ordinary citizens 
selected (rather than elected or self-selected) through near-random or stratified 
sampling. Once they are assembled, the citizen participants study an issue, delib- 
erate, then provide advice on a policy issue or proposal to broader publics, to 
elected bodies, or to executive agencies. Minipublics are typically created or under- 
written by an authoritative body, such as legislature, agency, or city council, and 
they supplement more familiar political processes. 


As is alluded to in this passage, a goal of this stage is typically to recruit a sample of 
participants who are representative of the whole constituency on features such as age, 
ethnicity, gender, education, income, etc. This is often accomplished through random 
sampling. 

Second, researchers then need to randomly assign participants to experimental 
conditions. In deliberative field experiments this often includes multiple treatment 
groups distinguished by varying features of the deliberative environment (Carman 
et al. 2015). For example, Neblo et al. (2018) randomize subjects into control, 
information-only, and full deliberation conditions to distinguish between informa- 
tion effects and deliberation effects per se. However, some researchers use repeated 
measures designs to assess the effects of deliberation. For example, instead of creat- 
ing a control group or multiple treatment groups as a basis of comparison, Luskin 
et al. (2002) compare participants’ attitudes after the deliberative event to the same 
participants’ attitudes prior to the event. 

There are advantages and disadvantages to using each design. The main strength 
of creating a control group or multiple treatment groups is that any significant dif- 
ferences between groups can be attributed to the effects of the deliberative event, as 
long as groups are balanced on covariates (Gerber and Green 2011, 7-8). Moreover, 
by using multiple treatment groups, researchers are able to distinguish the causal 
effects of interpersonal deliberation itself from the effects of other aspects of delib- 
erative forums such as providing information to participants (Carman et al. 2015; 
Esterling et al. 2011b). The major disadvantage of using a between-subjects design 
is that it may be difficult to balance on covariates, because relevant covariates when 
studying political phenomena include demographic variables such as race, gender, 
age, and education as well as political variables such as political interest, ideology, 
and partisanship (Kingzette 2021, 76). 

The main strength of using a within-subject design is that it ensures covariate bal- 
ance on these variables, meaning that any outcome differences cannot be attributed 
to differences in the composition of the samples. An additional strength is that 
within-subject designs increase statistical power by effectively doubling sample size. 
However, using repeated measures could lead to problems of internal validity, as it is 
difficult to know if changes in participants’ attitudes over time are due to the delibera- 
tive event itself, other events occurring in the broader political landscape, or response 
instability. An additional potential problem with within-subject designs is that they 
make it difficult to distinguish mechanisms of attitude change in experiments in 
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which there are multiple components such as expert panellists, informational pack- 
ets, and deliberation itself. However, scholars can pre-emptively address this problem 
by surveying participants before they engage in each step of the process (Farrar et al. 
2010; Hansen and Andersen 2004; Neblo et al. 2012). 

One option is also to use a between-subjects and within-subject design simultane- 
ously. This would involve creating a treatment and control group, but also measuring 
the attitudes of those in the treatment group at multiple times in the process. In a 
study of deliberative polling in the EU, Sanders (2012) was able to make between- 
subjects comparisons to confirm the causal effects of the event, but was also able to 
use within-subject comparisons to test potential mechanisms, finding that none of 
the proposed mechanisms (sampling bias, increased political knowledge, discussion 
quality, small group social conformity, and the influence of other actors) satisfacto- 
rily explain the attitude changes. While the best from an analytic point of view, the 
major disadvantage to this strategy is that it requires a large sample size and makes 
data collection more complex. 

Third, after assigning participants to conditions, researchers will need to invite 
participants to attend the deliberative event(s) and make these events happen. At 
this stage in the process, there are few commonalities across studies, but we believe 
it worth mentioning that these events often take vast resources. For example, to con- 
duct online Deliberative Town Hall events with members of Congress about a dozen 
times a year has required the assistance of several researchers, graduate students, and 
undergraduate students. 

Fourth, sometime after the event itself, researchers collect data from participants, 
as well as people in any control groups. This usually includes a survey that gathers 
participants’ attitudes, such as their opinions on the policy discussed in the forum, 
knowledge on the topic discussed, political efficacy, and reasoning behind their own 
opinions as well as those who disagree with them, and lagged effects of the treatment 
(Lazer et al. 2008; Lazer et al. 2015). Note that within-subject designs also require 
data collection from participants before the deliberative event, and often during the 
event as well. 


Pitfalls of Deliberative Field Experiments and How 
to Address Them 


At each stage of completing a deliberative field experiment, there are challenges 
that can threaten internal and external validity. When recruiting participants for 
a deliberative field experiment, one significant challenge is achieving representa- 
tiveness. The reason why this is such a significant challenge is that there may be 
attrition between initial recruitment and the event itself (Karjalainen and Rapeli 
2015), which represents a non-compliance problem. If non-compliance is correlated 
with any variables a researcher wants to balance on, then this can cause problems for 
inferring causal relationships by comparing averages across experimental conditions 
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(Esterling et al. 201 1a). This sort of differential non-compliance can be addressed in 
several ways. One option is to use large monetary incentives to strongly encourage 
all participants in treatment groups to participate in the deliberative event. Another 
option is to oversample constituents who are less likely to comply with treatment in 
the recruitment and randomization stages, so the sample of people who attend the 
deliberative event is approximately representative, even if those initially recruited 
are not. A final option is to accept differential non-compliance, but then use meth- 
ods of analysis that allow one to identify average treatment effects, even in the case 
of differential non-compliance (Esterling et al. 201 1a). 

A related challenge at the recruitment stage is recruiting a sizeable number of 
constituents into the study. Gathering enough participants is necessary to yield 
acceptable levels of statistical power to make inferences, but it is also often required 
for holding a successful event. From our limited observations, this challenge seems 
to have increased over the last several years, at least in the United States—survey 
response rates have plummeted (Keeter et al. 2017), people try to avoid members 
of the opposing party (Iyengar et al. 2019; Lelkes and Westwood 2017; Mason 2018), 
and trust in government institutions is at an all-time low (Pew Research Center 2019). 
One solution to this challenge is to provide financial incentives to participate or part- 
ner with research firms capable of recruiting large numbers of participants. Perhaps 
a simpler solution, however, is to identify the practices of deliberative field experi- 
ments that draw more constituents. For example, if researchers are recruiting through 
email, what subject lines and messages work the best? While these questions related to 
the marketing of deliberative events may not be of chief theoretical interest to most 
deliberative scholars, answering them may greatly assist in recruiting citizens into 
these studies. An additional way to address this problem is to use a within-subject 
design, which doubles statistical power, compared to a between-subjects design with 
the same number of participants. In some of our own research, we have turned to 
using a within-subject design precisely for this reason. 

In the randomization stage of a deliberative field experiment, one challenge will be 
to decide whether to use a between-subjects design, within-subject design, or both 
at the outset. Depending on which design a researcher uses, there are several related 
challenges. For researchers using between-subjects designs, perhaps the most dif- 
ficult aspect of randomization procedures is to achieve balance on covariates. One 
way to do this is to use block randomization. This involves creating several strata 
based upon combinations of covariates. For example, if it is important that treat- 
ment groups be balanced on age and education, we might create six groups of people 
who are: (1) college educated and between 18 and 40 years old; (2) non-college edu- 
cated and between 18 and 40 years old; (3) college educated and between 40 and 
65 years old; (4) non-college educated and between 40 and 65 years old; (5) col- 
lege educated and over 65 years old; and (6) non-college educated and over 65 years 
old. Then, within each of those groups, we would use a randomization procedure to 
assign participants to experimental conditions. This process helps to achieve balance 
on covariates when sample sizes are not sufficiently large to guarantee balance using 
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simple randomization techniques. Alternatively, researchers can correct for covariate 
imbalances by controlling for these covariates in statistical models. 

If researchers use a within-subject design, randomization will not be required, 
as every participant is in both a control and a treatment group. However, as men- 
tioned previously, the central challenge of using this approach is that it is impossible 
to know with certainty whether changes in participants’ attitudes from before and 
after the deliberative forum are due to the forum itself or other events in the broader 
political landscape. To mitigate against the concern that other factors cause pre-post 
changes, a couple of steps can be taken. First, holding deliberations on issues that 
are less salient makes it less plausible that external events change people's attitudes, 
and thus more plausible that the deliberative event causes any observed changes. Sec- 
ond, surveying participants closely before and after the deliberative event lessens the 
likelihood that observed changes are due to external events (Abernathy et al. 2019). 

The stage of deliberative field experiments in which researchers actually hold delib- 
erative events is likely the most arduous part of the entire process. It can involve 
ensuring participants find the location of the event, organizing multiple in-person or 
online sessions, bringing on moderators to lead discussion groups, inviting experts 
to speak on panels, ensuring technology is operating properly, and many other sub- 
stantial logistical hurdles. Depending on the size and scope of the experiment and the 
experience level of the researchers, the experiment might require a project manager 
or consultation with others who have successfully conducted such events in the past. 

After the deliberative event is over, participants will typically need to complete 
a post-survey. Just as in the transition between initial recruitment and attending a 
deliberative event, there is often attrition between the deliberative event and the post- 
survey. Indeed, in one ongoing study, we are finding that there is higher attrition 
in this stage of the process than between initial recruitment and the events, likely 
because in our initial recruitment strategy, we only invite people who are interested 
and able to attend the respective town hall to participate—people attend the online 
town hall to hear from their member of Congress, not to complete a (second) survey. 

If this attrition is correlated with relevant covariates, it could cause problems for 
external validity, as the study population will no longer be representative of the 
broader constituency. If this attrition is correlated with relevant covariates and exper- 
imental conditions, there is a deeper problem of internal validity, as treatment and 
control groups will no longer be directly comparable. One way to address this issue 
is to use weighting, a process that helps correct for unrepresentative samples by 
giving more weight to responses from under-sampled subgroups and less weight 
to responses from over-sampled subgroups. Alternatively, differential attrition at 
this stage can be handled through principal stratification methods (Frangakis and 
Rubin 2002; Horiuchi et al. 2007), though in this case ‘compliance type’ would indi- 
cate whether someone is predicted to attrit from the study between the deliberative 
event and data collection instead of whether someone is predicted to attend the ses- 
sion itself (i.e. ‘compliance type’ would not actually be compliance with treatment). 
Matching methods could also be used, which would involve comparing similar 
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participants across experimental conditions and then aggregating to identify treat- 
ment effects (Imbens 2015). However, all of these methods work best when there are 
high numbers of participants in each experimental condition. For studies with fewer 
participants, differential attrition can represent a serious threat to valid inference. 

Finally, we should also note that deliberative field experiments typically (if not 
always) violate the stable unit treatment value assumption (SUTVA), particularly 
dosage and spillover effects. SUTVA is violated if there is unmeasured variation in 
the dosage of the treatment and if one person's treatment affects another’. If we inter- 
pret deliberation as the treatment in such field experiments, then this assumption is 
almost never cleanly met. The whole point is that deliberating groups jointly create 
their own deliberative quality (dosage). Researchers have less experimental control 
because deliberative field experiments—by definition—are conducted ‘in the wild. 
For example, a participant in an online Deliberative Town Hall could be recorded 
as staying online the whole time but could be responding to emails instead of actu- 
ally listening. There is no method for researchers to induce all participants to listen, 
and attention is difficult to measure. In addition, for all deliberative field experiments 
with a discussion group component, the different discussion groups introduce vari- 
ation in the dosage of the treatment. While we can measure some aspects of these 
group dynamics (Bachtiger and Parkinson 2019; Neblo et al. 2012) it is infeasible to 
capture all potential variations. Thus, while we applaud efforts to remediate viola- 
tions of SUTVA in deliberative field experiments insofar as this is possible, we also 
want to make clear that at a certain point, this problem is intractable within this 
particular genre. The only way to manage the problem at a fundamental level is to 
combine results from a deliberative field experiment with those from a more con- 
trolled experiment—for example, one in which all but one of the ‘deliberators’ is 
really a confederate of the researcher with a script to follow. Even without combining 
different kinds of experiments, at least in some cases, the increased scientific general- 
izability and practical benefits of deliberative field experiments outweigh the decrease 
in internal validity compared to other study designs. 


Conclusion 


We have endeavoured to provide a conception of deliberative field experiments, 
summarize the types of research questions they are used to study, and review the 
processes and challenges of conducting them. We hope that the chapter will also 
function as a guide for scholars considering deliberative field experiments in think- 
ing through theoretical, analytical, and practical issues. In closing, we only want to 
add that conducting deliberative field experiments can be very challenging, but also 
incredibly rewarding both in terms of scientific insight and in facilitating democratic 
innovation and reform. 
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13 
Scenario Experiments 


Hannah Werner and Lala Muradova 


The systemic turn in deliberative democratic theory (Mansbridge et al. 2012) has 
expanded not only the breadth of research questions on deliberation, but also the 
methodological repertoire with which they can be studied. In this chapter, we present 
scenario experiments as a method that is suitable to study deliberation at two dif- 
ferent poles of the spectrum: micro processing in deliberation and macro effects of 
deliberative events on the wider public. In scenario experiments, people read a text 
or they hear and watch a video (a vignette), which is embedded within a survey. The 
vignette typically describes realistic deliberative processes using real examples and 
issues. At the same time, scenario experiments enable researchers to manipulate the 
variables of interest and precisely and randomly allocate respondents to treatment 
groups, which strengthens causal inference. 

It is not the goal of this chapter to provide a comprehensive introduction to the 
experimental method (for this purpose, see Chapter 11 by Grönlund and Herne) 
or scenario experiments in general because several excellent handbooks are already 
available (e.g. Druckman et al. 2011; Mutz 2011). Rather, our goal is to carve out 
its advantages for the study of deliberative processes. Scenario experiments cannot 
capture the complex interpersonal processes underlying a democratic deliberation, 
but they can be used as a complementary and powerful tool to study the processes 
which are either overlooked or impossible to disentangle by only studying real-world 
deliberation cases, especially when we look at the micro mechanism of internal delib- 
eration and at the macro effects of deliberative events. In the following, we will briefly 
introduce vignette experiments, and then discuss how they can facilitate deliberation 
research. We then present innovations in the field of experimental research that can 
be useful for deliberation research. Lastly, we present recent examples of scenario 
experiments in deliberation research and end with a discussion of its limitations. 


Scenario Experiments: A Brief Introduction 


Scenario experiments, also called vignette experiments, are a subcategory of sur- 
vey experiments. In general, survey experiments represent a marriage between two 
methods that are deeply rooted in the canon of the socio-scientific methods: large- 
N survey studies (see Chapter 14 by Gastil) and (previously mostly laboratory) 
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experiments (see Chapter 11 by Grönlund and Herne). The survey in ‘survey exper- 
iment’ refers to the use of survey sampling techniques to produce a large and 
heterogeneous sample (ideally representative of the target population, but not nec- 
essarily, since the causal inference is drawn from random allocation to treatment 
groups) on which the experiment is conducted. As Diana Mutz puts it, ‘it is the only 
kind of research design capable of simply and straightforwardly estimating popula- 
tion average treatment effects without complex statistical machinations’ (2011, 20). 
This is typically done through the use of online survey programming tools, such as 
Qualtrics. Experiment refers to the type of design where participants are randomly 
assigned to different treatment groups that vary on the independent variable (e.g. a 
citizens’ assembly is/is not convened to discuss a policy issue prior to a referendum). 
This clarification is important because it is different from the common use of the 
word ‘experiment’ in the deliberative field, where we often speak of a ‘deliberative 
experiment’ when talking about the (novel) implementation of a deliberative pro- 
cess (Gastil 2018). Likewise, the deliberative interventions initiated by researchers 
that include a pre- and post-design are thus not always experiments if they do not 
include random assignment to a treatment and a control group, or where the exper- 
imenter does not have full control over the manipulation (for a related discussion 
see Chapter 11 by Grönlund and Herne; see also Karpowitz and Mendelberg 2011). 
In recent years, survey experiments have become immensely popular in political sci- 
ence due to the possibility of programming them digitally, resulting in almost infinite 
possibilities for design with regard to the amounts and combinations of treatments 
or presentations of stimuli, and of filling in the surveys online, hence minimizing the 
costs of data collection and enabling a wider reach (Mullinix et al. 2015; Mutz 2011). 
Scenario experiments are based on a particular type of experimental stimulus. 
Respondents are typically asked to imagine a hypothetical scenario. Often, these sce- 
narios are described in short paragraphs of text (for alternative presentations, see 
the innovation section of this chapter). In deliberation research, this could be the 
description of a deliberative mini-public or a specific element thereof. Afterwards, 
respondents are asked questions about their perceptions of the described scenario, 
possibly including preferences, policy opinions, or assessments of legitimacy. 


What Scenario Experiments Can Do for Deliberation 
Research 


Deliberative democrats may be sceptical when considering the application of online- 
administered experiments to study deliberative processes. They may refer to a poten- 
tial mismatch between methodological assumptions underlying scenario experi- 
ments and normative theories of deliberation. Scenario experiments require a high 
level of experimental control with small manipulations of specific contexts that indi- 
viduals are exposed to. One might argue that deliberation is not what happens in 
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isolation but between people, in interaction with others (Dryzek et al. 2019). Online 
surveys with experimental stimuli, one might presume, can hardly create the same 
experience as an actual small-group deliberation. As Karpowitz and Mendelberg 
(2011) correctly suggest, experiments ‘may be particularly vulnerable to the disagree- 
ments between theorists and empiricists to the extent that their heightened levels of 
control bring more stylized and more artificial operationalizations of complex and 
multifaceted theoretical concepts’ (267). We agree that scenario experiments are not 
always suited to study interpersonal deliberation as a complex communicative and 
social phenomenon.’ But we want to make the case that scenario experiments can 
advance our knowledge about specific elements of the deliberative processes, par- 
ticularly when studying micro mechanisms of deliberation (see section on zooming 
in) or their connection to other spheres (see section on scaling up). Meanwhile, for 
those studying deliberative systems, scenario experiments can provide insights into 
causal relationships for isolated elements of a deliberative system, which should ide- 
ally be complemented with other methods, for instance, small-n methods that are 
more authentic and can account for the interconnectedness of elements. 

We identify two potential families of research questions that we consider crucial 
to our understanding of deliberation and deliberative systems and for the study of 
which scenario experiments are ideally suited. 


Research Area 1: Zooming In 


First, even though theorists and qualitative researchers rightly note that deliberation 
is more than the sum of its parts, understanding the role each individual factor plays 
in enhancing deliberative outcomes is crucial (Karpowitz and Mendelberg 2011; 
Mutz 2011). This is particularly important for a systemic understanding of delib- 
eration. Most scholars have moved away from considering only those interactions 
that fulfil high normative ideals as deliberative and towards embracing more less- 
than-perfect forms of deliberation. The argument is that these forms of deliberation, 
although imperfect on their own, could altogether contribute to larger deliberative 
systems. Thereby, gaining an understanding of causal inference in terms of the role of 
individual characteristics of deliberation has become a crucial task. This is especially 
the case for the study of individual cognitive processes in deliberation, sometimes 
referred to as deliberation within or reflection (Goodin 2003; Muradova 2020). Thus, 
tracing back the assumptions and goals of deliberative theory to micro processes 
of cognition has brought deliberation research ever closer to political psychology 
(see also Karpowitz and Mendelberg 2018; Muradova 2021; Myers and Mendelberg 
2013). We consider scenario experiments as relevant tools for research because they 
enable: (a) more exact and controlled manipulation of key features; (b) cleaner causal 


* Experimental designs do not, however, preclude us from recreating this kind of deliberation in 
subjects’ imagination, via scenarios (we discuss this more in the sections that follow). 


Hannah Werner and Lala Muradova 193 


inferences about their effects; and importantly (c) data collection across a heteroge- 
neous group of respondents who go beyond the usual suspects who may participate in 
deliberative mini-publics. For example, if we want to study whether personal narra- 
tives within interpersonal deliberative settings can foster more engagement amongst 
underrepresented groups, for example citizens with lower levels of formal education 
or women, we can use scenario experiments. Although the scenario experiment in 
this case would not entirely approximate the real-world deliberation, it can give us 
some idea about the role and effect of these communicative elements for citizens’ 
political reasoning and deliberation. 


Research Area 2: Scaling Up 


A second area of research question relates to recent expansions of the focus from what 
happens inside the deliberative minipublics to the embeddedness of such forums in 
the wider democratic system (e.g. Curato and Boker 2016; Fung 2015; Lafont 2017; 
Niemeyer and Jennstal 2018). Upscaling of deliberative events is a crucial task for 
deliberation scholars and practitioners. Since deliberative forums are typically rather 
small-scale, it is not enough to study whether they can have effects for the handful of 
people that are involved directly in these processes. To understand the role that delib- 
erative events can play in democracies at large, their potential to affect the broader 
public is pertinent. Two linkages are of importance. 

Deliberative forums can impact the wider public by shaping citizens’ policy deci- 
sions by either motivating citizens to become interested in the issue and to get 
informed about it or by directly informing their preferences (Goodin and Dryzek 
2006). Following Warren’s (2017) account of a problem-based approach to demo- 
cratic theory, a core function that can be fulfilled through deliberation is collective 
will formation. This alludes to deliberation in a broader sense than deliberative 
mini-publics, but it is still consequential for the role that mini-publics might play: 
stimulate collective will formation, not only amongst the participants but also the 
wider public (see Gastil 2018; Gastil et al. 2016; Ingham and Levin 2018; Suiter et al. 
2020a). The case of the Irish Citizens’ Assembly and its role in shaping public prefer- 
ences with regard to the issue of abortion is an excellent example (Suiter 2018). For 
instance, research has found that statements from the Citizens Initiative Review, pre- 
sented in the context of a scenario experiment to the subjects, substantially improved 
knowledge about the issue (Gastil et al. 2016). 

Deliberative forums are also often expected to foster perceived legitimacy of 
authorities and decisions that result from deliberative consultation. Again, while the 
legitimacy perceptions of the participants of the mini-publics are worth investigat- 
ing, the bigger challenge is to boost perceived legitimacy amongst citizens who did 
not personally participate in these processes, who are mere observers of the process 
(Curato and Boker 2016; Goodin and Dryzek 2006; Pow et al. 2020). 

For the study of both these outcome variables of interest, we make the case 
that scenario experiments are suitable. They provide information cues that mimic 
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information that citizens would encounter in the real world about the existence, pro- 
cedural character, and outcome of a deliberative forum. At the same time, they allow 
for the exact manipulation of the type and content of information and the conditions 
under which the desired effects on policy opinions and legitimacy beliefs can occur. 


Using Scenario Experiments in Deliberation Research 


We identify five advantages of scenario experiments in complementing the existing 
methodological canon in deliberation research. The obvious advantage of any exper- 
imental approach is that it enables inferences about causal relationships. Since this 
major advantage is already discussed elsewhere (e.g. Mutz 2011), we will not address 
it here and will instead focus specifically on the advantages of scenario experiments. 


Disentangling the Relative Role of Individual Design 
and Context Factors 


While laboratory or field experiments have many other advantages, they usually con- 
strain the number of independent variables that can be studied in conjunction. This 
has to do primarily with efforts related to introducing new treatments and, probably 
more importantly, with small sample sizes that are typical of ‘offline’ experiments. 
Survey experiments, in contrast, allow for the manipulation of a range of specific 
design and contextual factors of the deliberative process. This not only increases our 
knowledge of the individual role of these factors in achieving deliberative outcomes, 
but also further provides insights into their combined effects (Mutz 2011). Scenario 
experiments enable us to study a new range of research questions. For instance, is it 
more important for citizens’ perceptions of legitimacy that deliberative forums are 
demographically representative on the national level rather than on the local level? 
Or are different designs desirable depending on the conflictual nature of the policy 
issue in question? 


Increasing Experimental Realism 


A challenge to all experimental studies is achieving high levels of external validity. 
The same holds true for experiments on deliberation. Often, laboratory experiments 
create highly abstract decision-making situations, such as the division of money 
between members of a group, that are in many ways detached from actual polit- 
ical processes in the real world (see, for instance, Dickson et al. 2008; Sulkin and 
Simon 2001). Other times, laboratory experiments involve citizens discussing a pol- 
icy issue in small groups (Muradova 2020), but the results of these discussions are not 
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connected to realistic policymaking. Scenario experiments can depict processes that 
mimic policy decision-making processes in the real world. The freedom and flexibil- 
ity of the method allows for modelling of scenarios that directly connect to processes 
and policy debates in the region or community under study. Issues can be selected 
based on ongoing public debates at the time of data collection or on real recom- 
mendations made by an existing deliberative organ (see, for example, Suiter et al. 
2020a). Lastly, online survey tools allow for individual adaptations of the scenario, 
for instance, by inserting the name of the respondent’s municipality into the text. 
When thinking particularly about the emerging research agenda on the macro 
effects of small-scale deliberation processes, scenario experiments can realistically 
convey similar amounts and types of information as citizens would receive in the 
real world. Stimuli could, for instance, take the form of media coverage, such as 
newspaper articles or television clips (see, for example, Suiter et al. 2020b). 


Going beyond the Usual Suspects: Representative Samples 


A key advantage of survey experiments, as outlined above, is the combination of 
causal inference methods with diverse and heterogeneous sampling techniques. The 
opportunity to study the relationships of interest amongst representative or at least 
heterogeneous samples of the population holds two key advantages for deliberation 
research. First, a prominent concern amongst deliberative democrats and their critics 
is that deliberation and deliberative forums are primarily appealing to a specific set of 
people—those who are already highly interested in politics, hold high levels of formal 
education and socioeconomic status, and belong to ethnic and cultural majorities 
in society.” Since these groups already have access to political power through rep- 
resentation, providing them with extra channels for policy input may actually be a 
bad idea, so the argument goes. Irrespective of whether this critique actually holds 
empirically (for counterexamples, see Neblo et al. 2010), scholars should take these 
concerns seriously when studying deliberation. Studying participants’ views, experi- 
ences, and preferences alone will give us only an incomplete picture of deliberation. 
Following Karpowitz and Mendelberg’s (2011, 268) recommendation that ‘how those 
who were not part of the discussion understand deliberating groups is a topic worth 
considerable additional study, we emphasize here that survey experiments facili- 
tate the study of people who would not usually turn out to a deliberative event (as 
well as those who would). By posing an extremely low barrier for participation (fast, 
practical, flexible), survey experiments can enable us to study groups that usually 
refrain from all things political.? However, one must be sensitive to other types of 


> Alternatively, other groups of people, although not necessarily unwilling to participate in these 
processes, may be faced with other, structural inequalities. 

° Of course, the actual representativeness of the sample depends on the sampling technique, which 
in turn depends on the agency that recruits respondents. This can range from market research compa- 
nies (cheap, but they usually sample from their own pre-recruited subject pool and not from the whole 
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exclusion that may occur through this method, such as Internet access that may 
be less available to older people or in regions of the world where access is more 
restricted. 

Second, the opportunity to obtain representative samples of the population allows 
for the study of heterogeneous effects across subgroups. One strand of literature 
within deliberative theory points out the potentially exclusionary and elitist nature 
of ‘rational arguments. To remain inclusive of all groups in society, the acknowl- 
edgement of other types of communication styles was called for, such as personal 
stories, narratives, or other forms of emotional speech, as part of the repertoire of 
legitimate reasoning techniques (e.g. see Sanders 1997; Young 1996 for the initial 
critique; Bachtiger et al. 2010 for a recent perspective). These theoretical debates 
can be enriched and complemented by survey experimental research. For instance, it 
could be tested whether certain types of reasoning indeed speak to different demo- 
graphic groups better and thus can serve as measures to make deliberation attractive 
to, and useful for, everyone. Accordingly, survey experiments can be important tools 
in studying how to overcome inequalities in deliberation. 


Thinking about Statistical Power 


A common although little-discussed problem in quantitative deliberation research 
is that of statistical power. Simply put, power describes the probability of obtaining 
significance values that reject the null hypothesis (below the chosen alpha level, for 
instance.05) when the alternative hypothesis is true. Thus, the more power a study 
has, the less likely it becomes to falsely accept the null hypothesis (to make a Type II 
error). Power is a function of the size of the effect and the sample size of the study. 
This means, particularly when studying small effects, that sufficiently large sample 
sizes are required to detect these effects. Indeed, as in most social science research, 
deliberation research usually yields small effects. At the same time, actual deliberative 
events often involve only a small number of people, hence they struggle to achieve 
sufficient levels of power (usually considered above 80 per cent). Survey experiments 
with bigger samples and enough power thus provide the opportunity to obtain valid 
and reliable estimates of causal relationships. 


Last but Not Least: Costs 


While there is great value in conducting real-life deliberative experiments, they are 
also immensely costly. Survey experiments cost only a fraction (depending on the 
sampling method), both in terms of the price of data collection itself and of time and 


population) to panels set up for research practices, such as TESS at the University of Pennsylvania (more 
expensive or application based, but often use probability samples drawn from registry data). 
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personnel. This has two important implications for deliberation research. The first 
relates to the questionable generalizability of findings obtained from countries with 
high research resources, the so-called WEIRD (Western, educated, industrialized, 
rich, and democratic) countries. By developing low-cost tools to conduct delibera- 
tion research, we can open the doors to scholars working in countries with limited 
resources (e.g. in the Global South but also Southern and Eastern Europe). In this 
vein, low-cost survey experiments also facilitate comparative research projects, with 
the same experiments being run in several regions of the world, which are to date 
rare in deliberative research. Second, as outlined by Mutz (2011), the affordability 
of survey experiments makes it possible to spread resources across several studies, 
opening up avenues for replication and follow-ups on interesting research findings. 
As such, they can contribute greatly to cumulative and reliable knowledge creation 
in deliberation research. 


Illustrative Examples 


In this section, we showcase some examples of how survey experiments have been 
used in the study of deliberation. 


Micro Processes in Deliberation 


Strickler (2018) conducted two scenario experiments (one online and another 
telephone-based) to examine the extent to which individuals’ adherence to delib- 
erative discursive norms is conditioned by their partisanship. More specifically, he 
tested the effect of partisan social identities on citizens’ attitudes to reciprocity when 
exposed to disagreement. Subjects in his scenario experiment were first asked about 
the extent and dimensions of their partisan attachment. Further, individuals were 
asked to indicate their position on two policy issues—immigration and civil liber- 
ties. The subjects were exposed to a counterargument, which was attributed to either 
a Democrat, a Republican, or a person without a party identifier. After the stim- 
uli, respondents were asked to answer a set of questions tapping into the concept of 
reciprocity. The results of the research indicate that party cues moderate the rela- 
tionship between disagreement and citizens’ attitude of reciprocity. In other words, 
subjects in the experiments were more likely to consider in-party disagreement with 
an open mind while they were less likely to do so for out-party disagreement. Thereby, 
by researching two crucial components of deliberation—exposure to disagreement 
and reciprocity—with the help of a scenario experiment, the author could examine 
one of the conditions under which citizens are more (or less) willing to adhere to 
deliberative discursive ideals. 
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The Effects of Deliberative Mini-publics on Opinion 
Formation amongst the Broader Public 


Ingham and Levin (2018) conducted scenario experiments in the United States to 
investigate whether knowledge of deliberation outcomes can stimulate citizens to 
update their opinions on policy issues. Their experiment was included in a national- 
representative online survey. Participants first read a short introductory text about 
the potential changes to social security on different issues. Afterwards, they were 
randomly assigned to one of the issues and received either no text (control group) 
or different texts that described the occurrence of a citizen assembly, resulting in 
different recommendations and different reactions by a congressional panel. The 
respondents were then questioned on their opinions on the policy proposals. The 
authors find that, while information cues about a mini-public and their recommen- 
dation can increase support for the policy proposal, this is not the case for all issues. 
Further, it seems that hearing about the advice by a mini-public prompts people to 
doubt their initial policy preferences by increasing their uncertainty about the poli- 
cies. This experimental approach allowed the authors to study citizens’ reactions to a 
simple deliberative cue under different conditions. By including two different policy 
issues in the design, they were able to detect differences in effects, depending on the 
issue at hand. 


The Effects of Deliberative Mini-publics on the Legitimacy 
Perceptions of the Broader Public 


Boulianne (2018) similarly studied the macro impacts of small-scale deliberation 
processes, also focusing on perceptions of legitimacy. She conducted two scenario 
experiments that were connected to real ongoing deliberative processes in Edmon- 
ton (climate change) and Alberta (energy efficiency) in Canada, that had received 
little media attention. Both experiments had two conditions: a control condition in 
which respondents did not receive any information, and a treatment condition in 
which respondents read a short text. The vignettes described the mini-publics’ pro- 
cesses and policy recommendations. After reading the paragraph, respondents were 
questioned about their policy preferences, trust in government in the respective pol- 
icy areas, and political efficacy. The findings differ between the two studies, which 
can be due to several factors, such as the different policy issues or other features of 
the process. This study illustrates the trade-offs that scholars face when designing 
scenario experiments. While using a real case increases external validity, it also con- 
strains the freedom to manipulate elements of the process or the recommendations 
of deliberative minipublics, which would consequently help uncover a more precise 
understanding of the mechanisms. 
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Survey experimental research is an exciting field with many innovations in the 
design, fielding, and analysis of experiments. Some of these innovations are fruitful 
for deliberation research. First, as the previous examples have shown, the challenge in 
web-based experiments is the design of clear and realistic stimuli that explain often- 
unknown deliberative processes to respondents while maintaining a high level of 
attention and engagement. One potential solution to this is to present the stimuli 
with the help of audio-visual material rather than text (for an example, see Werner 
and Marien 2020; Suiter et al. 2020b). This can improve citizen engagement with the 
content, increase attention, and clarify and emphasize the important pieces of infor- 
mation.* This mode can also approximate the real-world information flow for many 
people, who mostly find out about local and national events by watching television. 
One straightforward and comparatively cost-efficient approach is to create animated 
movies with voice-over text. Multiple tools to create such movies are available online 
and require only basic knowledge of image and sound editing. While we consider 
this a fruitful innovation for deliberation research, studies that compare the effects 
of texts and audio-visual material are needed. 

Second, the increasing use of conjoint experiments expands the possibilities for 
designing scenario experiments (Hainmueller et al. 2014). Conjoints simultaneously 
test multiple factors, resulting in a number of conditions (often above 100), which is 
unthinkable in standard (laboratory) experiments. Since the factors are orthogonal to 
each other, their independent effect can be assessed across all other independent vari- 
ables, which can help ensure generalizability. There are two ways in which conjoints 
can be useful for the study of deliberation. First, they allow for the simultaneous study 
of multiple elements that characterize deliberative processes, such as the composition 
of the participant group, effective influence, the structure of the assembly, the role of 
politicians in the process, and so on. Second, conjoints can increase generalizabil- 
ity because they investigate one or a few variables of interest while taking contextual 
factors into account. Hence, the individual effect of, say, the participant recruitment 
method for deliberative events could be tested across contextual factors, such as dif- 
ferent levels of government or policy issues. First studies such as Christensen (2020) 
or Goldberg and Bachtiger (2022) made use of conjoint analysis to study citizens’ 
preferences for different design components of a participatory process. 


Challenges of Scenario Experiments 


So far, we have demonstrated the advantages of scenario experiments. We con- 
clude our chapter by discussing some of its limitations. First, external validity and 


* Some may argue that this type of exposure makes it difficult to make a cleaner causal claim, due to the 
impossibility of controlling for the effect of different visual cues. 
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generalizability must be assessed critically with every scenario experiment. Even if 
experimental stimuli are designed in a realistic fashion, they are still presented in the 
context of a survey and hence detached from everyday political life (Barabas and Jerit 
2010). To date we lack reliable systematic knowledge about whether and under which 
conditions the results obtained with scenario experiments match findings from real 
cases, particularly because it is unclear what the benchmark would be (e.g. compar- 
ing the findings to a cross-sectional survey study might be inconclusive since none 
of the two studies can credibly claim to hold authority over ‘true’ effects). Existing 
research remains ambiguous about the generalizability of scenario experiments, with 
some studies finding supporting evidence (e.g. Hainmueller et al. 2015; Mullinix et al. 
2015) and others drawing more pessimistic conclusions (Barabas and Jerit 2010). 

A second concern is that some scenario experiments may fail to meet the infor- 
mation equivalence assumption, crucial for experiments, designed with the aim of 
studying the real-world effects of some factor or attribute, presented as an infor- 
mational piece. In other words, it is assumed that when reading about the vignette, 
citizens’ beliefs about background characteristics of the treatment are not affected. 
This is an important assumption, particularly for experiments aimed at examining 
epistemic effects, that is, ‘the effects of changing subjects’ beliefs about some factor of 
interest, holding constant beliefs about background features of the scenario (back- 
ground beliefs)’ (Dafoe et al. 2018, 400). This is less of a problem when studying 
informational effects where the experimental treatment and the naturally occurring 
treatment are very similar (as is the case with the macro-research questions described 
previously). To the extent that this assumption holds true (citizens think about a 
group of different people coming together to talk about an important issue in an 
open, equal, and tolerant environment, when reading about the word deliberation), 
we are talking about the real effects. To the extent that it does not (citizens shift their 
beliefs, for instance, because they thought of a group of highly educated, middle-aged, 
male citizens coming together to show off how smart they are), the effect obtained 
can be easily attributable to some other aspect of the manipulation. Accordingly, the 
internal validity of the experiment can be questioned. Several strategies have been 
proposed to ensure information equivalence, such as manipulation checks (see Dafoe 
et al. 2018). In addition to these suggestions, we propose another strategy, perhaps 
more relevant to deliberation research. We believe scenario experiments can embed 
a short definition of the treatment—the concept we are trying to manipulate in the 
experiment—within the vignette. For example, if our treatment is a mini-public as 
a decision-making body, we could provide the respondents with a short definition 
of what a mini-public is and what it entails. In this way, we can prevent them from 
updating their beliefs about the term ‘mini-public’ and keep the definition constant 
in their reasoning. 

Despite the above-mentioned challenges, scenario experiments provide us with 
unique opportunities to disentangle and test the effects of different types, dimen- 
sions, and components of public deliberation under different conditions and contexts 
amongst larger and more heterogeneous populations. The lessons learned from 
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these experiments contribute not only to the advancement of deliberation research, 
but also to the innovation of institutional design to improve the functioning of 
democracies. 


Further Reading 


Auspurg, Katrin, and Thomas Hinz. 2014. ‘Introduction to Conjoint Experiments. In 
Factorial Survey Experiments, Vol. 175. London: Sage. 

Mutz, Diana. 2011. Population-Based Survey Experiments. Princeton, NJ: Princeton 
University Press. 
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Survey Methods 


John Gastil 


Social scientific surveys are essential in research on deliberation because they provide 
the most reliable means of measuring and aggregating public attitudes. Deliberative 
theory addresses the development of public judgement, the wax and wane of demo- 
cratic legitimacy, and the public’s willingness to weigh opposing views. Surveys are 
essential to measuring all of these. 

Surveys have been used effectively to both challenge and buttress the premises of 
deliberative democracy. For example, Hibbing and Theiss-Morse (2002) used survey 
data to show that Americans want government to act on their behalf, rather than 
asking them to deliberate on complicated issues. This finding bolstered scepticism 
about deliberation until a more precise study of public attitudes found that people 
were willing to engage when given a realistic expectation of having influence (Neblo 
et al. 2010; Neblo et al. 2018). 

Survey methods also provide the foundation for the most recognizable reform 
inspired by deliberative democratic theory. ‘Mini-publics’ gather a randomly selected 
microcosm of the public to study and comment on policy questions, and they typi- 
cally use survey methods to recruit their participants. For many mini-publics, surveys 
also measure opinion shifts within the deliberative body (Fishkin 2018) or trace their 
wider influence on the public at large (Mar and Gastil 2019). 

In this chapter, I will examine these and other uses of surveys as a research tool 
by deliberation scholars and practitioners. I begin by reviewing the basic theoretical 
assumptions of this method, then review methods of data collection and analysis. 
Afterwards, I provide strong examples of surveys conducted with the participants 
in deliberative events, followed by studies that survey larger populations. In the 
conclusion, I consider how surveys might help advance the theory and practice of 
deliberation in the future. 


What Surveys Assume, Reasonably and Otherwise 


Broadly defined, a survey asks a respondent to answer a series of questions then 
repeats that process across a large enough number of people so that one can gen- 
eralize its findings to an even larger population. To make this aggregation possible, 
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most survey questions are closed-ended, which means that respondents must choose 
their responses from a list provided. 

Although many of the assumptions underlying surveys have been challenged for 
reasons that are important to deliberative theory, researchers have found ways to 
mitigate these problems.’ First, deliberation requires the inclusion of diverse voices, 
and critics worry that surveys produce biased population samples or cause people 
holding unpopular views to censor themselves when responding. Indeed, surveys 
must attend to both hazards of random sampling and social desirability bias, but 
these can be managed by setting response quotas for (or even oversampling) histor- 
ically marginalized social groups and through careful question phrasing that invites 
unpopular responses and reassures respondents of their anonymity. 

A more fundamental assumption in surveys is that people hold real attitudes and 
can express them cogently by responding to survey questions. In fact, researchers 
have found that people sometimes express opinions when they have none. These 
‘non-attitudes’ are given to satisfy the interviewer's request for an answer, or they can 
reflect inattentiveness to the survey itself (Alvarez et al. 2019; Saris and Sniderman 
2004). Thus, a respondent might oppose a non-existent law or express a view about 
something they do not grasp, such as nanotechnology. Even genuine responses to a 
policy attitude question constitute a reaction to particular words, rather than to the 
objects those words are meant to represent. Thus, respondents may identify them- 
selves as ‘conservative’ or ‘liberal’ to express their feelings about those exact words, 
rather than any underlying ideology those words purport to measure. 

Surveys address these problems in several ways. For example, to reduce the weight 
of particular words in a survey, researchers use multi-item scales and require rea- 
sonably consistent responses to make inferences about respondents’ underlying 
attitudes. To capture elusive non-attitudes, researchers often preface questions to 
emphasize this option (e.g. ‘or do you have no opinion one way or the other?’). 

More generally, researchers can understand their surveys as a conversation 
between respondent and interviewer rather than a direct measurement of attitudes. 
In this view, a person’s response to survey items amounts to a quasi-public statement 
of what they believe, which often translates into behaviours, such as voting in elec- 
tions. After all, voters sometimes back a populist candidate or vote for a referendum 
to ‘send a message; even though they do not support the policies the candidate or 
referendum represents. 

Finally, surveys assume the validity of subjective assessments that respondents pro- 
vide about their experiences, such as when they must rate the quality of a deliberative 
process in which they participated. These subjective accounts, however, do not always 
accord with objective measures or content analyses of the same deliberative events 
(Gastil et al. 2012). Some forms of recollection are less problematic than others, but 
the most defensible position is that subjectivity is important. After all, who is in a 


* For a balanced critique of the survey enterprise and its history, see Herbst (1993). For a detailed 
description of survey methods, see Dillman et al. (2009). 
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better position to report on whether a respondent felt respected or silenced dur- 
ing a deliberation—the participant or a neutral third party? It may be useful to have 
multiple measurement methods, but in cases such as these, participants’ subjective 
accounts matter. 


Survey Data Collection and Analysis 


When studying public deliberation, it is easiest to separate surveys into two 
varieties—surveys of specific deliberative bodies and broader public surveys. These 
two types of survey present different challenges and call for different methods of data 
collection. 


Sampling Methods 


For deliberative bodies of citizens such as mini-publics and juries, one can aspire to 
and achieve exceptionally high survey response rates (e.g. 70 per cent and higher). 
After all, the population that one aspires to survey has already self-selected as being 
responsive, whether to a jury summons or a less formal invitation. Surveys of par- 
ticipants in deliberative events are typically conducted on paper or with online 
applications (via mobile devices, laptops, etc.), but mail, phone, or Internet surveys 
may precede or follow such events. 

For broader surveys, attaining higher response rates presents a greater challenge, 
and obtaining a statistically representative sample of a diffuse population can be 
costly. Even for the venerable Pew Research Center, phone response rates have 
collapsed—all the way down to 6 per cent in 2018 (Kennedy and Hartig 2019). 
Turning to online surveys can still provide broadly representative samples, but only 
when paying the higher price required to access professionally maintained panels 
of respondents. More common are ‘nonprobability samples’ of online respondents, 
provided through a lower-cost commercial vendor. Even when one must use demo- 
graphic quotas to align a survey sample with a population’s census parameters, 
such samples are useful for survey experiments, which benefit from diverse samples 
without requiring representative ones. 


Measurement 


Surveys used in deliberation research typically measure demographics, policy atti- 
tudes, policy-relevant knowledge, political orientations, and other variables relevant 
to hypotheses, such as personality traits, previous political experiences, etc. Surveys 
of participants in deliberative events also often include retrospective assessments of 
the deliberative experience itself. None of these question sets are standardized within 
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the deliberation literature, and researchers commonly turn to other literatures for 
specific survey items. 

A vexing problem for survey researchers studying deliberative events concerns the 
measure of deliberation itself. This should not surprise those familiar with the mul- 
tiplicity of theoretical definitions for the term ‘deliberation (Bachtiger et al. 2018). 
In practice, survey measures of deliberative experience generally have measured two 
aspects—the rigor of a deliberative body’s policy analysis and the democratic social 
quality of its discussions. 

The 2009 Australian Citizens’ Parliament provides a useful example because it used 
both observer ratings and survey measures (Gastil 2013). On the two main days of the 
process, trained observers rated the analytic rigor of the process somewhat favourably 
(5-5.6 on a scale from 1-9) and gave higher ratings to its democratic social process 
(6.2-6.9 on the same scale range). This finding generally accorded with participants’ 
assessments: 76 per cent of respondents said that their fellow citizens ‘stated positions 
without justifying them’ only ‘occasionally’ or ‘never, and 85 per cent felt that other 
participants treated them with respect ‘almost always. 

There also exists a diversity of approaches to measuring the policy knowledge 
and attitude changes that occur as a result of deliberation. The most straightforward 
approach uses single-item measures before and after deliberation to track change, 
but repeated measures at different stages of a deliberation can illuminate more clearly 
when participants shift their beliefs (Farrar et al. 2010). Multi-item measures of policy 
attitudes can also reveal the degree to which participants have adopted more coher- 
ent clusters of attitudes as a result of deliberating (Gastil and Dillard 1999; Niemeyer 
and Dryzek 2007). A more demanding approach codes the content of open-ended 
responses to measure the full repertoire of arguments that a participant develops 
on an issue (Cappella et al. 2002), which can show how deliberation helps people 
consider multiple perspectives. 


Unit of Analysis and Statistical Power 


Surveys of deliberative bodies face a challenge common to all studies of human 
groups—the tension between the individual and the group as the focal unit of 
analysis. For example, a public opinion researcher might draw a sample of 400 
individuals to create a large sample with enough ‘statistical power’ to detect even 
small differences in attitudes amongst respondents (Cohen 1988). If a sample is too 
small, it cannot reliably test hypotheses because any failure to produce a statistically 
significant result could be explained by the study’s inadequate sample size. 
Deliberation scholars employing surveys often work with large samples of indi- 
viduals, but these respondents are rarely independent observations. Instead, the 
respondents to these surveys are often interacting with each other in smaller groups. 
For example, recent studies of deliberative events have attempted to disaggre- 
gate the effects of individual differences versus group/network-level differences in 
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participants’ experiences (Bonito et al. 2014; Christensen et al. 2016; Farrar et al. 
2010; Suiter et al. 2014; Tucker and Gastil 2013). Deliberative events are often singular 
affairs, as in the case of a mini-public where the sample size is one (i.e. one mini- 
public). Studies measuring group-level effects typically resort to comparing breakout 
sessions within such events. This often results in a modest group-level sample size 
(e.g. 150 people broken into groups of 5 has a group-level N = 30). 

To appreciate the severity of this problem, consider the cautionary tale of the 
most ambitious randomized controlled trial in this field of study (Carman et al. 
2015). A team of investigators sought to measure the impact of four alternative 
deliberative methods on public knowledge and attitudes towards evidence-based 
medicine. Nearly a thousand study participants participated in seventy-six twelve- 
person groups across the US, with another 377 in a control group that read materials 
without discussion. Survey measures showed results generally in the predicted direc- 
tion, but with small to moderate effect sizes observed at the group level of analysis, 
few of the observed differences reached statistical significance. 

As a general guideline for comparative studies using group-level analyses, 
researchers need to have fifty groups for each deliberative method. This provides 
sufficient statistical power so long as the researcher posits directional hypotheses 
and expects a moderate effect size, such as when predicting that Method A will out- 
perform Method B on some survey metric.” If the expected effect sizes are smaller, 
the necessary sample size doubles or triples, and non-directional research questions 
require even larger samples. To date, no large-scale comparative study has met this 
standard for adequate statistical power at the group level of analysis. These demand- 
ing guidelines for group-level sample size often lead researchers to study smaller 
laboratory groups that meet only briefly (Gastil et al. 2008; Karpowitz and Mendel- 
berg 2014). Unfortunately, it can be difficult to generalize from these groups to 
more intensive mini-publics that have professional facilitators and more meaningful 
political contexts. 


Panellist Survey Examples 


To appreciate the variety of survey purposes and methods, the remainder of this 
chapter provides examples of studies focused on either (a) the participants in deliber- 
ative events (‘deliberative panellists’) or (b) wider publics. When studying panellists, 
the three most common approaches are to conduct a survey before, during, after, 
or long after a process. Surveys taken beforehand establish a baseline, whereas 
those taken after a deliberation either look for change over time or comparisons 
across groups. Surveys during an event can measure unfolding processes, whereas 
those taken long after a process concludes can assess long-term impacts or panellist 
reflections. 


? For details on how to make such power calculations, see Cohen (1988). 
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Even those researchers who have access to a deliberative panel before it begins its 
work sometimes hesitate to record baseline measures of attitudinal variables for fear 
of anchoring panellists to those pre-deliberation beliefs. This worry is more than the- 
oretical. Baccaro et al. (2016) found that students in deliberations on immigrant civil 
liberties were less inclined to change their opinions if they stated their views on a 
pre-deliberation questionnaire. This baseline survey was taken just before delibera- 
tion began, but that practice is common for low-budget processes that do not attempt 
to collect random samples of participants far in advance of a deliberative event (Gastil 
and Dillard 1999). 

The Citizens’ Juries process represents a mini-public that eschews baseline sur- 
veys (Crosby and Nethercutt 2005) out of a concern for anchoring attitudes. It 
provided the blueprint for the Oregon Citizens’ Initiative Review, which uses a strat- 
ified sample of twenty to twenty-four citizens to analyse a ballot measure and write 
a one-page voting aid that gets distributed to the full electorate. The absence of a 
pre-deliberation attitude measure has forced researchers to ask panellists to report 
their initial opinions retrospectively at the end of their multi-day deliberations (Gastil 
et al. 2015), which raises concerns about the validity of any self-reported attitude 
changes. 

In experimental settings, however, post-only measurement has a strong justifica- 
tion when random assignment permits the assumption of equivalent pre-deliberation 
scores for participants in control and treatment groups. A recent study on the 
behavioural impact of deliberative discussion took this approach and found no signif- 
icant effects of discussing public assistance programmes on subsequent willingness 
to take political action (Myers et al. 2018). In cases such as this, which combine care- 
ful design with ample statistical power, the omission of a pre-deliberation measure 
of the dependent variable strengthens causal inference because it removes the risk of 
the initial measurement biasing results. 


Pre- and Post-Deliberation 


More commonly, researchers take baseline measures of attitudes before and after a 
deliberative process. This is the hallmark of Deliberative Polling (Fishkin 2018), but 
it is also common practice in experimental forms and other mini-publics (Grönlund 
et al. 2009). The greatest challenge in a pre- and post-deliberation survey method is 
the risk that the content of the deliberation will veer far afield from the particular 
questions asked during the survey. This is the most straightforward explanation for 
why so many participants’ opinions do not change, even after participating in multi- 
day discussions (Merkle 1996). In cases where the deliberative topic and discussion 
were closely aligned, as in the case of Texas polls on state energy policy, Deliberative 
Polls have produced dramatic shifts in opinion (Fishkin 2009). 
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As more research accumulates, it may turn out that the direction of aggregate 
change in deliberative events follows a pattern that transcends issue and context. For 
example, looking across the poll items that the Center for Deliberative Democracy 
reported as showing significant change, deliberating appears to promote more cos- 
mopolitan (versus nationalist) attitudes (Gastil et al. 2010). A more recent study of 
an EU-wide deliberative poll found similar directional shifts amongst participants as 
compared with a parallel control group (Sanders 2012). 


Repeated Measures Mid-Process 


To measure how participants experience a deliberative event, some studies employ 
multiple measures at different intervals during the process. The aforementioned 
Oregon Citizens’ Initiative Review includes a survey at the end of each day that 
includes roughly one dozen items concerning that day’s deliberations. This permits 
researchers to see how the participant experience might ebb and flow over the course 
of a process. Each day at the Review, for example, the panellists answered this ques- 
tion: “How important a role did you play in today’s panel discussion?’ Over the course 
of four days, the average response on a scale that ranged from 1-5 increased from 
roughly 3 (‘Moderately important’) to 4 (“Very important’) for all five of the Reviews 
held in 2014 (Gastil et al. 2015). Similar approaches have measured longer-term 
deliberative dynamics that stretched across successive weeks or months, rather than 
merely consecutive days (Blais et al. 2008; Karpowitz and Raphael 2014). 


Longitudinal Effects 


Other panellist surveys take a longitudinal approach but stretch their survey over 
a longer period of time to see the long-term impacts of participating in delibera- 
tion. A representative example is a study on climate change policy in Edmonton, 
Alberta (Boulianne 2018). Between their initial selection in September 2012, and a 
follow-up survey six months later, deliberative panellists showed an increase in their 
general political trust and their internal political efficacy. When a final follow-up 
was conducted two years later, however, both effects had disappeared. By contrast, 
domain-specific trust (municipal policymaking on climate change) and a perception 
of system responsiveness showed a significant increase only in that final follow-up 
survey. 


Retrospective Change and Causal Attribution 


A less common survey method asks participants to look back on their deliberative 
experience to discern whether they believe the deliberation changed them. A study 
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using this approach to study both the Australian Citizens’ Parliament and the Ore- 
gon Citizens’ Initiative Review found a consistent pattern across these two distinct 
mini-publics (Knobloch and Gastil 2015). In surveys conducted a full year after delib- 
eration, participants reported changes in their attitudes towards deliberation (e.g. 
were more inclined to agree that ‘people from different parties can have civil, respect- 
ful conversations’). When participants were asked if ‘attending the [Parliament or 
Review] has caused you to change the frequency with which you do the following 
activities, large percentages reported that deliberation made them more active in 
many ways, except for conventional political participation (e.g. ‘going to political 
meetings’ or ‘volunteering for parties or candidates’). 


Public Survey Examples 


Theories of deliberative democracy encompass far more than forums and mini- 
publics, with a renewed emphasis on the more systemic level of analysis (Parkin- 
son and Mansbridge 2012). To understand system-level dynamics, deliberation 
researchers can survey large populations and trace changing deliberative norms, atti- 
tudes, and behaviours. Large-scale survey experiments can offer insight into how the 
public uses information to form opinions and make voting choices. Some of these 
surveys link discrete deliberative forums and large-scale elections to understand how 
people use the information provided by mini-publics when taking policy positions 
of their own (Warren and Gastil 2015). 


Population Trends 


Population surveys can supplement existing political surveys to ask respondents 
about deliberative experiences and attitudes that are not measured in conventional 
survey designs. Jacobs et al. (2009) did this with a comprehensive survey of adult 
US residents. The investigators argued that taking deliberative theory seriously 
requires discerning what percentage of people attend public meetings (25 per cent) 
or engage in any kind of ‘discursive participation, including political conversations 
(81 per cent). Their survey delved into the details of these encounters to understand 
people’s motivations and experiences of deliberation, and it set the stage for future 
research to record variations in survey responses over time and across nations. 


Awareness and Assessment 


Surveys can also determine the impact of mini-public deliberation on broader 
publics. This was clearest in the case of the British Columbia Citizens’ Assembly, 
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which recommended that the province change from a first-past-the-post voting sys- 
tem to a variant of single-transferrable-vote (STV). A survey found that the ‘STV’ 
terminology was alien to the average British Columbian. Thus, voters’ assessments 
of the proposal depended partly on learning about the Citizens’ Assembly’s delibera- 
tions and trusting the judgement of its members (Cutler et al. 2008). Amongst those 
who knew nothing about STV, 58 per cent supported the proposed voting system 
if they knew about the deliberative Assembly that recommended it. By contrast, only 
51 per cent of those unaware of the Assembly’s role supported the proposal. After all, 
nearly two-thirds of respondents (65 per cent) agreed with the statement, “Because 
the Citizens’ Assembly members are people like me, I trust their judgment. 

Influence aside, it is useful to know what proportion of the public knew anything 
about the Assembly: In the year leading up to the election, public awareness rose from 
a low of 40 per cent to more than 55 per cent (Cutler et al. 2008). By comparison, 
telephone surveys of Oregon voters from 2010 to 2016 show that its awareness of 
the Citizens’ Initiative Review process held every other year in that state rose from 
40 per cent in 2010 to 52 per cent in 2012. It rose to 56 per cent in 2016—but no higher 
(Gastil et al. 2017). More detailed analyses make clear that much of this awareness 
lacks detailed understanding about the deliberative process and that voter trust may 
be conditional on having such knowledge (Gastil et al. 2018). 


Survey Experiments 


Whereas the British Columbia case traced voter awareness and attitudes towards a 
mini-public, other studies have used survey experiments to estimate more directly 
the impact of information about these deliberative bodies. These surveys provide a 
randomly selected subset of respondents with snippets of information about a mini- 
public’s process and its findings; they then use statistical comparisons to discern 
whether those exposed to this information answer subsequent questions differently 
than those respondents not provided with such information. Learning a mini- 
public’s findings can alter people’s policy views, depending on the issue and audience 
(Boulianne 2017; Ingham and Levin 2018a, 2018b; Mar and Gastil 2019). A similar 
design found that more detailed information about the Citizens’ Initiative Review 
could alter not only policy attitudes but also voting choices (Gastil et al. 2018). 


Conclusion 


Looking forward, survey methods will continue to aid deliberation researchers in the 
coming years, and knowledge in this field will advance more rapidly if future research 
improves on previous efforts in several respects. More consistent measures of deliber- 
ation across different discrete events will make comparisons more meaningful, as will 
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the inclusion of a standard set of input and output measures. The survey attached to 
the Participedia online archive aspires to meet this aim, but it has yet to become a 
standard assessment tool. Cross-validation of deliberative surveys with observational 
and content analytic methods will bolster confidence in the utility of survey mea- 
sures. If incorporated into randomized controlled trials (Carman et al. 2015) with 
necessary statistical power at the group level of analysis, such research could clarify 
precisely what aspects of deliberation connect to which outcomes (Gastil et al. 2017). 

Finally, surveys can do more to inform systemic theories of deliberation. For 
example, researchers can trace feedback loops from convening specific deliberative 
events back to public trust in government (Boulianne 2018). More broadly, does the 
institutionalization of mini-publics change how citizens and public officials behave 
or view each other? A cross-national survey of legal experts, for example, found an 
association between governmental effectiveness and the use of lay participation in 
the judicial branch, such as through criminal juries (Voigt 2009). Hopefully, future 
surveys of citizens, civic leaders, and public officials will tell us more about how the 
introduction (or erosion) of deliberative practices influences complex democratic 
systems. 
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Social Network Analysis (SNA) is a toolkit, widely used in the social sciences, to 
model the interdependence of the components that make up a social system. The 
main advantage of this approach is the visual representation of the flows that are 
present in social exchanges: be these markets or deliberative processes. This approach 
can be employed if the relational nature of the phenomenon is available to systematic 
observation (Lazega and Higgins 2014). 

In this chapter, we outline how SNA can be used in the study of deliberative 
processes. Overcoming the old dichotomy that considers agents and structures as 
separable entities, SNA assumes that agents and structural contexts are combinatory 
effects of social processes (Granovetter 1985; White 2008). In this way, it provides 
a lens for the identification and examination of such processes, allowing the mea- 
surement or modelling of their effects on collective life, including the processes of 
group formation through solidarities and exclusions, collective learning and social- 
ization, emergence of influential leaders, political coalitions, belief formation, social 
controls and conflict control and, last but not least, the processes of regulation and 
institutionalization (Lazega 2012). 

There are many ways scholars of deliberative democracy can utilize SNA in their 
research. For example, SNA can help us to identify the central and peripheral actors 
of a debate taking place in a deliberative arena like parliament, or the public sphere. 
It shows us which actors have the opportunity to influence the decisions inside a 
particular arena. As such, it helps us to investigate the normative assumptions of 
deliberative theory, such as the participatory equality or deliberative equality of par- 
ticipants. Utilizing SNA, researchers can measure the deliberative inequalities of 
power and influence. They can also study the deliberative systems and investigate 
whether and how a deliberative system works, whether different components of the 
system relate to each other, whether and how discourses and ideas flow from one site 
to another, and the extent to which some clusters of discourses are isolated from the 
rest of the system. 

In this chapter, we first present types of networks and some metrics used in SNA. 
In the second section, some articles are introduced as examples of SNA application 
in the study of deliberative processes. In the third section, we focus on two-mode 
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networks, presenting a case that used SNA to measure the pathologies of delibera- 
tive systems. Lastly, the conclusion presents a synthesis of SNA contributions and 
limitations to the study of deliberation. 


Types of Networks and Metrics in SNA 


We have selected four cases as good examples showing how SNA can help opera- 
tionalize some of the key questions in deliberative democracy research. This selection 
follows several methodological criteria that a beginner in SNA should take into 
account. First, we present the examples of studies that work with one-mode delib- 
eration networks. This type of network structure is characterized by the adjacency or 
relationship between agents of the same nature (people, groups, political parties, etc.) 
(Figure 15.1). This is followed by the presentation of a study that analyses a two-mode 
deliberation network. This type of structure is characterized by the incidence of an 
agent of one ontological level, ordinarily people or social groups, with an event that 
constitutes an occasion for interaction, such as an assembly, a discussion forum, or 
a ritual celebration. In Figure 15.2, the mode of events is represented by squares and 
the mode of actors by circles. 

In each demonstrative case, we use several criteria for a cross-sectional analysis: 
(1) the deliberative problem under study; (2) the operationalization of the problem 
in terms of a set of structural variables measurable through a graph (i.e. the visual 
representation of a deliberative interaction network); and (3) the type of data and 
methodology of collection. 


Figure 15.1 One-mode Network 
Note: Legend: circles = actors. 
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Figure 15.2 Two-mode Network 
Note: Legend: squares = events; circles = actors. 


There are three basic dimensions that describe a network structure, and 
each dimension requires a different kind of metrics for their operationalization 
(see Table 15.1). 

Network cohesion designates the connectivity of the social system, that is, the 
extent to which all agents are connected to each other. It is usually used to investigate 
the connectivity of members with each other in a social setting. For example, a low 
density means that members of a social system are not engaged in social exchanges. 
Focusing on network cohesion, Resende (2018) studied the participation of private 
and civil society representatives in all thematic commissions of Brazilian Congress 
and compared the models of core and periphery using the densities of two-mode 
matrices as criteria of differentiation. 

Actor centrality expresses the relative importance of actors in terms of their status 
and influence over the entire system. For example, this is representative of power 
understood as balanced or unbalanced recognition between social actors (Blau 1964). 


Table 15.1 Dimensions and Metrics of Social Networks 


Dimensions Metrics 
1. Network cohesion Density, Distance 
2. Actor centrality Nodal degree (in or out), Ego-node, Ego-net 


3. Cohesive subgroups Clique, Islands 


Source: elaborated by the authors. 
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As such, it is relevant for studies on participatory institutions or mini-publics because 
it allows us to identify the central actors within these deliberative arenas. 

‘Cohesive subgroup’ refers to connectivity across the network. In empirical 
research it is used to identify the formation of coalitions or other kinds of solidarities 
between its members. Investigations can show the strength and scope of the connec- 
tions within subgroups in a given network, and this is relevant to the comprehension 
of power relations and the capacity to act. Joschko and Glaser (2019) provide an inter- 
esting illustration of the study of a cohesive subgroup when they investigate Twitter 
discussions in Germany to map actors who strongly supported quotas of represen- 
tation for women. These dimensions and metrics describe one-mode and two-mode 
networks. 

The first step, for anyone interested in learning about and applying SNA in their 
research, involves familiarization with the metrics used in this type of analysis. It is 
possible to define eight types of metrics: 


(1) Density: the ratio between observed relationships and the possible total; since 
it is a proportion, the scale ranges from 0 to 1. 

(2) Distance: the number of relationships between two nodes in a network. 

(3) Ego-node: an actor that is taken as a reference point in the analysis of a set 
of relations; it constitutes the centre of a partially analysed network, as con- 
sequence from any ego-node we can estimate the out- and in-degree which 
argue to be defined bellow (See number 5 and 6). 

(4) Ego-net: a sub-graph that takes an ego-node as a reference point, including 
the relations with all its peers and the relations of the peers to each other. 

(5) In-degree: the total of arcs, or oriented arrows, entering in a node; this is a 
metric of an actor’s popularity. 

(6) Out-degree: the total of arcs, or oriented arrows, exiting a node; this is a metric 
of actor’s initiative. 

(7) Clique: this is a subgroup, within a network, identified by the total adjacency 
between nodes; the minimum size of a clique is a triad where all actors are 
related to each other. 

(8) Island analyses: this technique considers the intensity of the relationships 
between the arenas that compose a subsystem in order to identify parts of the 
network connected by stronger inner ties than the ties outside the subgroup, 
called islands. 


This section presented the difference between one-mode and two-modes networks, 
introducing a few relevant metrics used in SNA. Such metrics are important for the 
comprehension of the selected examples in the following two sections. We will start 
with a few examples of studies employing one-mode network analysis in the study 
of deliberation. We will then move to an investigation employing two-mode network 
analysis. 
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One-mode Network Analysis in Deliberative 
Democracy Research 


One-mode network analysis can be useful to deliberative scholars wanting to map 
relationships between agents across deliberative systems, indicating the structure of 
the system and the role played by different actors. They can be particularly useful for 
deliberative scholars seeking to identify the main actors of a given arena, for example 
a parliament or a social media site such as Twitter, based on the degree of centrality of 
actors and/or groups in these arenas. One-mode network analysis enables researchers 
to examine the strength of the ties between different actors and/or groups which then 
might help with the comprehension of deliberative processes. It is possible to iden- 
tify the strength of the ties between the actors whose support the executive proposes 
inside the parliament. In order to do this, we can see the position of the actors inside 
the network, and the strength of the ties established between them with regard to 
some specific subject matter. The examples below illustrate how one-mode network 
analysis can be employed for the study of deliberation. 

The first example presents one of the first efforts to use SNA to study deliberative 
process. Manlio Cinalli and Ian O'Flynn (2014) applied SNA to study the political 
deliberation process in relation to the integration of Muslims living in the United 
Kingdom. The research question investigated was as follows: if Muslim communities 
adequately enter into the process of political deliberation, do they raise their chances 
of integrating into the community of rights and obligations? Data was extracted from 
a selection of articles published in The Guardian and Times newspapers, representing 
Labour and Conservative views on the issue of Muslim integration in the UK. 

The study was operationalized by two analytical dimensions: (1) claims making 
analysis; and (2) political integration in relational terms. The first dimension anal- 
ysed the content of the newspaper articles with three deliberative metrics, namely: 
(a) acceptable language; (b) supporting arguments; and (c) appeals to the general 
interest. The second dimension was relational and sought to measure the ties of sup- 
port and the ties of dissent from other groups towards Muslims. The SNA metrics 
used were the following: (a) density; (b) distance; (c) ego-node; (d) out-degree; (e) in- 
degree; (f) ‘cliques. Figure 15.3 indicates an extensive presence of relationships of 
mutual support in the field. If we assume Muslims as an ego-node, we can observe that 
they are immediately supported, that is, it is at distance d = 1, by government, civil 
society organizations, and executive agencies. By contrast, Muslims are not directly 
supported by pro-minority actors and political parties, that is, it is at distance d > 1 
(Cinalli and O'Flynn 2014). 

The second example of one-mode analysis is the study of deliberative systems in a 
large scale in the United States of America. Using this kind of analysis, Barvosa (2018) 
analysed how public opinion in the US became broadly supportive of the rights of 
LGBT people. The data was generated in an online search on the topic between 2007 
and 2015. Different kinds of documents were gathered, such as videos, music, films, 
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Figure 15.3 Ties of Support 


Note: Density=0.23. 
Source: Cinalli and O’Flynn (2014, 441). 


and news in the media. The methodology consisted of a qualitative analysis of the 
documents. Three structural factors of the deliberative system were operationalized: 
(a) deliberative entrepreneurs; (b) deliberative packages; (c) precipitating events. The 
ego-node metric was used to demonstrate how deliberative entrepreneurs actions 
enabled outreach, on an effectively national scale, for the advocacy of LGBT peo- 
ples rights. These entrepreneurs carried out actions of dissemination because they 
were embedded in a wide network of LGBT actors and leaders, dispersed over the 
US territory. 

Finally, the third example of one-mode analysis is the research about the relation- 
ship between the representative claim expressed in parliament and its validation in 
public opinion (Twitter) using Social Network Analysis. Joschko and Glaser (2019) 
studied, from the point of view of representation, the quotas for women in the 
German parliament. Two main questions were posed: (1) Who are the central actors 
on the issue of quotas for women in the German parliament? and (2) How are 
the speeches made in parliament validated on Twitter? The data source was the 
parliament’s tachygraphic notes, by means of which it was possible to analyse and 
categorize the speeches, given by parliamentarians, in favour of, against, and neutral 
with regard to quotas for women. Then, the Twitter debate was analysed to identify 
the central actors in the discussion. SNA was used to measure the validation of those 
speeches by public opinion. To do so, the network of tweets that were focused on quo- 
tas was reconstructed. Through the island analysis’ technique, the network sectors 
with actors more densely connected to the discussion about quotas were identified. 


1 Island analyses: this technique considers the intensity of the relationships between the arenas that 
compose a subsystem in order to identify parts of the network connected by stronger inner ties than the 
ties outside the subgroup, called islands. 
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Two-mode Network Analysis in Deliberative 
Democracy Research 


Two-mode networks are particularly appropriate for the study of deliberative sys- 
tems, because they allow us to identify which actors are involved in the discussion 
of issues carried out in specific deliberative arenas. Amongst the many possibili- 
ties of employing two-mode networks (such as the study of networks formed in 
parliaments), we will present a particular application of a method called Affiliation 
Network Analysis. In this technique, it is assumed that actors, whether individuals or 
groups, operate with an intrinsic mechanism of networks, coupling and decoupling 
(White 2008). That is, the co-participation of agents connects events, and, inversely, 
taking part in the same events connects agents. By co-participation, we refer to the 
same actor being present in two or more different arenas. Co-occurrence is the 
presence of two or more actors in the same arena. This approach offers a valuable 
methodological innovation for the empirical study of political coalitions in deliber- 
ative systems. In what follows, we explain this methodological approach through an 
example of how it can be used in practice. 

We argue that two-mode networks can be used to diagnose pathologies that hinder 
a functional deliberative system. Deliberative systems are described as dysfunctional 
when, amongst others, they are unresponsive to a diversity of discourses or when 
they prohibit new discourses from emerging (Parkinson and Mansbridge 2012). SNA 
can meet the challenge of understanding the precise character of these dysfunctions. 
Below we outline some pathologies that can be identified using this method. In each 
pathology, we suggest more specific tools (Da Silva and Ribeiro, 2016a; Da Silva, 
Ribeiro and Soares, 2016b; Da Silva and Ribeiro, 2017). 


Tight Coupling within the Deliberative System 


These are situations in which parts of the system are so strongly connected to one 
another that it is difficult to make way for dissenting views or prompt reflexivity. 
According to the two-mode approach, when considering members’ simultaneous 
participation in different arenas, the contexts in which the areas of the system share 
a large number of members, with a high degree of overlap, indicate greater approxi- 
mation between the arenas. An extreme case of this pathology would be a system in 
which all the arenas are composed of the same members. Tight coupling within the 
deliberative system can be examined by evaluating the degree of network integration 
through the analysis of density, proximity, and distance between the actors. 


Decoupling within the Deliberative System 


This is the opposite of tight coupling. Decoupling refers to the degree of dissociation 
amongst different parts of the system, which, in turn, hinders ideas, proposals, or 
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reasons from travelling from one part to another. SNA can be used to determine this 
by subgroup analysis. Techniques of analysis of subgroups in social networks allow us 
to identify such subsets of connected deliberative areas. The different techniques are 
distinguished from each other based on the criteria used to delimit the subgroups, 
namely: (1) the mutuality of ties which identifies parts of the network in which all 
actors in a subgroup have ties between them; (2) the accessibility of members, which 
locates parts of the network where the members of subgroups are at a distance n from 
the others’; (3) the frequency of ties between members, which focuses on subgroups 
in which the actors have an x proportion of their changes within the subgroup; (4) the 
relative frequency of ties between members and non-members, whereby subgroups 
are identified according to the intensity of the ties between their members. 


Institutional Domination 


This is a situation in which one part of the system exercises mastery over the others. 
From the point of view of Social Network Analysis, this situation can be detected 
by analysing centrality measures. The node degree is the basic metric that allows for 
the identification of hierarchies in a network structure. For example, it is possible to 
detect how popular an actor is by measuring the number of arcs that arrive at her/him. 
It is also possible to detect her/his entrepreneur level by measuring the number of 
arcs that depart from her/him. Through these techniques it is possible to identify 
several aspects that reveal positions with the potential to influence the exchange of 
resources in a network. Network analysis tools allow one to identify powerful actors 
in a network as those with a greater number of direct relationships with other actors, 
or as those who act more often as intermediaries between other actors or connect to 
their peers over shorter distances. 

The analysis of the above-mentioned pathologies can be done through the inves- 
tigation of participation in different arenas of discussion. Data can be collected 
from attendance lists at meetings, in minutes, news articles, and other documents. 
Once collected, this information can be organized in an incidence matrix’ in which 
any actor, named formally as n, and any event m is coded and classified based on 
co-participation and co-occurrence criteria. Actors can be people or organizations. 
Events are occurrences that connect actors. Co-participation refers to events in which 
two or more agents play a part, while co-occurrence refers to actors sharing two or 
more events. Software programs such as Pajek and Ucinet can help to analyse the 


? We can think of this distance as the maximum number of intermediaries that two actors in the same 
group need to interact, defined by n-1. 

° In the case of one-mode matrices, being incident means that two actors have a relationship with each 
other. This is expressed in the structure of a matrix by marking a binary or scale value, in vertices where 
the line of actor n with an actor j is located. In the case of two-mode matrices, the incidence means the 
participation or presence of an actor in an event. The registration of the observed incidence follows the 
same logic as in the one-mode matrix. 
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data and generate graphs, visually representing co-participation and co-occurrence. 
In doing so, we focus mainly on how actors and events are related. 

In what follows, we present our operationalization of the two-mode network 
approach to diagnose systemic pathologies in a specific case. We have investigated 
a system of participatory arenas in the municipality of Belo Horizonte (Brazil). The 
first step of the research was identifying the deliberative arenas. We focused on one 
type of participatory arena, which is central in Brazil: councils. Since the early 1990s, 
councils have been the main Participatory Institutions (IPs) in the country, hosting 
deliberations to define strategic policy guidelines (Avritzer 2009; Avritzer and Souza 
2013). Councils are participatory innovations that were first introduced in 1988 by 
the Brazilian Constitution and by ordinary laws, which regulated specific articles of 
the Magna Carta. The Brazilian political system requires states and municipalities to 
create councils on key policy areas such as health, social work, and youth issues. Part 
of their deliberations concerns how funds from the federal government are used. 
Members of the councils are selected based on elections amongst members from 
NGOs, which are responsible for representing civil society. The President, governors, 
and mayors appoint the counsellors who represent the government. In general, coun- 
cils have around thirty members. The meetings are open to members of the public 
who can comment during the deliberation process, although they cannot vote in the 
final decision-making process. 

Figure 15.4 illustrates the application of the two-mode approach in the system of 
councils in the city of Belo Horizonte, Minas Gerais, Brazil. 

From the point of view of connectivity, it can be observed that different arenas 
(squares) vary in size (number of actors participating), but that, in general, the system 
tends to connect almost completely, which is represented by ties (lines) amongst the 


Figure 15.4 Two-mode Network, Municipal Councils Subsystem of Belo 


Horizonte 
Note: Legend: square = councils; circle = actors participating. 
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different circles in the diagrams. Figure 15.4 shows the deliberative system in the 
form of a two-mode network, which allows us to see the role of the actors (circles) in 
the connection of the arenas. The lines in the figure show the affiliation relationship 
between actors and deliberative arenas. 

The system is fully integrated. We note the presence of more central arenas in 
the system because they share more counsellors. These counsellors exert influence 
in multiple arenas and may thus, hypothetically, dominate the process. Our analy- 
sis of municipal council subsystems in Belo Horizonte revealed that five councils in 
the middle of the figure could dominate deliberative process; however, we also noted 
that there is no isolated arena. This might suggest interactions amongst deliberative 
arenas that favour the transit of ideas, values, and decisions despite some institutions 
being more central than others. 

In order to understand what leads to the emergence of systemic pathologies, it is 
necessary to investigate the ties between deliberative arenas. Figure 15.5 shows the 
connection between the systems arenas. SNA software offers algorithms that allow 
one to transform two-mode networks into one-mode networks. This is a resource for 
manipulating the original data which allows one to visualize the same information 
in a new way. A two-mode network can give rise to two one-mode networks: (1) a 
network of actors, represented in dots, and linked by lines that represent the number 
of events that participate together; and (2) a network of events, represented by dots, 
and linked by lines that represent the number of actors they share. The figure below is 
a network of events, since the deliberative arena is where the actors meet to deliberate. 
The different widths and shading of the lines (light grey to black) refer to the strength 
of the tie between the arenas, considering the number of actors they share. 


Figure 15.5 One-mode Network, Municipal Councils Subsystem of Belo 
Horizonte 
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Our analysis suggests a trend towards minimal institutional domination in the sys- 
tem. The centrality of the arenas is relatively well distributed: degree centralization 
= 0.352. This metric indicates the propensity of an actor in the network to central- 
ize power. The closer to 1, the more the network structure is organized around a 
single central actor. The closer to 0, the greater the number of central actors. This 
suggests that the system is decentralized, thus making institutional domination dif- 
ficult. Based on this, we can assume that there is a flow of communication between 
the arenas by facilitating the transmission of ideas from one space to another. 

This flow of communication varies depending on the positions of each arena inside 
the network. The observation of peripheral and central positions of actors within a 
network is a spatial measurement to determine domination. Central positions are 
occupied by arenas that have many relationships with other arenas, while peripheral 
positions are occupied by arenas with little or no relationships with other arenas. 
Central arenas are more likely to exercise institutional domination. In the figure 
above, it is possible to differentiate the condition of the arenas by comparing the 
size of the points. 

As noted above, the arenas are connected by ties of different intensities. Arenas that 
share more counsellors are more likely to be connected by a flow of information. Note, 
for example, the presence of the three strongest relationships between the deliberative 
arenas (black line). In the case of the municipal system, the proportion of ties with a 
value greater than 1 is most common and represents 42.5 per cent of observed ties. 
This indicates that almost half of the relationships between the arenas of the system 
share at least two actors. 

SNA also enables us to undertake a subgroups analysis based on the intensity of 
the connection between the arenas (Wasserman and Faust 2009; Nooy et al. 2011; 
Higgins and Ribeiro 2018). This strategy allows for an evaluation of tight coupling and 
decoupling situations, as it reveals the existence of subgroups, as well as the intensity of 
the connections between arenas and the subsystem. In our research, we have analysed 
subgroups based on islands analysis. This technique considers the intensity of the 
relationships between the arenas that compose a subsystem in order to identify parts 
of the network connected by stronger inner ties than the ties outside the subgroup, 
called islands. The researcher must stipulate the minimum and maximum size of the 
islands to define the number of actors to be included in the subgroups. The maximum 
number is only completed when all the actors that are connected to the group by 
lesser intensity do not exceed nine actors. If this occurs, the number of arenas closest 
to the defined maximum for the subgroup, and connected by the same intensity of 
the island, will then be selected. 

Figure 15.6 shows the islands analysis performed for the system of councils in 
Belo Horizonte. The transmission of communication between arenas is more likely 
to occur between islands. The size of the circles shows the degree of connection of 
one council with all the others. Larger circles represent, therefore, councils that share 
more counsellors within the subsystem. The intensity of these ties is indicated by the 
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Figure 15.6 Islands in the Deliberative Subsystem of the Belo Horizonte Policy 
Councils 


Source: Developed by authors based on data from Belo Horizonte City Hall. 

Note: Key: white = disconnected councils; gray = island of councils related to the theme of 
food; dark gray = island of urban policies; soft gray = island of policies of human rights and 
social policies. 


widths and colour intensity of the lines in the grey scale. Wider and darker lines 
represent strong connections. 

In this study, we identified three islands. There is a small island with two coun- 
cils (grey), made up of the municipal councils of food and nutritional safety and the 
school nourishment council. The relationship between the two policies is evident. 
These councils share four counsellors. In this subgroup, the size of the squares shows 
that the first arena is more connected to the subsystem, albeit weakly, since it shares 
few members with other arenas of the subsystem. 

A second island is found on the right of the figure in dark grey. It is composed 
of councils generally associated with urban policies. It brings together the municipal 
councils for economic development, housing, sanitation, urban mobility, environ- 
ment, urban policy, and cultural heritage. A striking feature of this island is the 
central position that the urban mobility council occupies, which shares the most 
members with other councils within this subgroup. 

The third island of the subsystem is formed by a set of councils linked to human 
rights policies and social policies (soft grey). The presence of a sector organized 
around the youth council on this island could reflect a conception of youth policy, 
which directly involves sport, culture, and the promotion of racial equality poli- 
cies. On the other hand, the councils on human rights policies are organized around 
the municipal council of social assistance, where a strong relationship was observed 
between it and the municipal councils of rights of the child, of the adolescent, and of 
the elderly. 

Five weakly connected municipal arenas (white) in the deliberative subsystem can 
also be observed. In this case, the arenas appear isolated, as no strong ties with other 
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arenas have been identified. This means that despite sharing counsellors with other 
parts of the subsystem, the number of shared members was not sufficient for the 
arenas marked in grey to integrate any of the identified islands or to compose new 
islands. 

Island analysis is useful for investigating the structures that are hidden at first 
glance. In the illustrated case, it helped us to reveal that despite the general connec- 
tivity that was initially apparent, Belo Horizonte’s deliberative system of councils is 
structurally vulnerable to the pathologies of decoupling and institutional domina- 
tion. The system was divided into three subgroups, in addition to the presence of 
isolated arenas. And it was possible to identify arenas centralizing the relationships 
within the islands. As such, it is necessary to study the groups inside the network 
because some deliberative assumptions might not be present in some arenas. 


Conclusion 


In this chapter we sought to illustrate how SNA techniques can help us in the empir- 
ical study of deliberative democracy. We differentiated between different forms of 
network analysis—one-mode networks and two-mode networks—which researchers 
can draw on to investigate issues at the heart of the deliberative agenda, including 
asymmetries, domination, brokerage, and insulated discussions. Our two-mode net- 
work research on councils in the municipality of Belo Horizonte (Brazil) sought to 
show how the approach allows us to comprehend whether and how different subsys- 
tems are connected, while also diagnosing pathologies that may hinder the flow of 
ideas and proposals across a system. Introducing key dimensions and metrics used in 
SNA, the chapter also illustrated how network graphs can help in the characterization 
of systems, providing visual cues to the comprehension of discursive processes. 

It is also important to acknowledge some of the limitations of the SNA method. 
One limitation of SNA, as applied to deliberative systems, has to do with the difficulty 
of operationalizing human interactions mediated by discourse, because discourses 
are not physical objects moving from one entity to another. Human communication is 
a complex process, and one does not express or receive messages in the same way that 
one buys and sells goods. Mapping these abstract and multidimensional processes 
can be tricky, which will pose some challenges to deliberative scholars using SNA. 


Further Reading 


Hanneman, Robert A., and Mark Riddle. 2005. Introduction to Social Network Meth- 
ods. Riverside, CA: University of California (published in digital form at http:// 
faculty.ucr.edu/~hanneman/). 
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Higgins, Silvio Salej, and Antônio Carlos Ribeiro. 2018. Análise de Redes em Ciências 
Sociais. Brasilia: Enap. 

Nooy Wouter de, Vladmir Batagelj, and Andrej Mrvar. 2011. Exploratory Social 
Network Analysis with Pajek. New York: Cambridge University Press. 

Wasserman, Stanley, and Katherine Faust. 2009. Social Network Analysis: Methods 
and Applications. New York: Cambridge University Press. 
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16 
Big Data Analysis 


Nuria Franco-Guillén, Sebastian De Laile, and John Parkinson 


Scholars of deliberative democracy are faced with the challenge of researching an 
overwhelming abundance of complex, system-wide communication, following the 
systemic turn (Parkinson and Mansbridge 2012). Deliberative systems cover a wide 
range of practices that may not be deliberative if assessed in isolation, but which 
may serve deliberative functions when viewed in relation to their interactions with 
other parts of the system, including ‘everyday talk around kitchen tables and other 
informal settings (Mansbridge 1999; Neblo 2015). The challenges of researching 
such systemic communication are enormous (Fleuß et al. 2018), and most of the 
methodological strategies deployed so far are unsatisfactory. 

Big data analysis can help researchers access and make sense of a mass of commu- 
nicative detail. Big data is sometimes defined as data that is too large for a user-level 
computer to process; but for the social sciences a better definition is that it combines 
high-volume, fast-paced production, and loosely structured natural language. This 
is exactly the kind of data that deliberative systems scholars are interested in: the 
millions of everyday interactions produced in the online public sphere. 

In what follows, we outline the key issues and choices that need to be made with 
big data analysis. We do not simply describe those choices in general terms, partly 
because much depends on the aims of the research, and partly because it is a new 
and rapidly changing field, with little agreement on how to do it or prior application 
in deliberative studies. New possibilities and applications are being invented daily. 
Instead, we offer guidance on how to deal with common challenges in big data analy- 
sis by sharing insights from our own research. That research generated and compared 
two big databases on two ‘national conversations’: the first comprising a million 
online interactions over two years before and just after the Scottish independence 
referendum of 2014; the second 100,000 interactions on a campaign to ‘Recognise’ 
Aboriginal and Torres Strait Islander peoples in the Australian constitution.’ The 
cases were explicit attempts to generate mass public deliberation. 

The chapter begins by situating big data analysis in the context of existing meth- 
ods of deliberative systems research. We then discuss data collection and analysis by 
revealing the steps we took and the choices we made, including the presentation and 


1 For details on the Scottish independence debate, see Keating and McEwen (2017) and Parkinson et al. 
(2020); for a discussion of the Recognise campaign from an agonist perspective, see Maddison (2017). 
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analysis of results. The discussion reveals much about the pros and cons of working 
with big data sets, but also the insights that can be gained. 


Big Data and Deliberative Systems Research 


Deliberative systems theory emphasizes the importance of myriad venues of recog- 
nizably political conversation in the informal public sphere, where citizens engage 
and act critically with each other (Dryzek 2010), but also share narratives of every- 
day experience of collective life (Bachtiger and Parkinson 2019). This is the ‘pool of 
perspectives’ (Bohman 2012) which helps ground and legitimate formal deliberation 
(Parkinson 2006). 

The problem is that our ability to examine that pool of perspectives is limited 
because of the classic breadth/depth trade-off: sacrificing communicative detail for 
the sake of a structural overview, or vice versa. There are several ways of handling 
this in the deliberative literature. The first and dominant strategy is to focus on com- 
munication within and between one or two mini-publics and a formal institution or 
decision moment (e.g. Boswell et al. 2016), a strategy that tends to focus on ‘delib- 
eration’ rather than the more dispersed, emergent quality suggested by Mansbridge 
et al. (2012). The second strategy is the opposite: focus on the big picture of institu- 
tional networks and draw inferences about communication based on those linkages 
(e.g. Cinalli and O'Flynn 2014). The third strategy focuses on discourses as coordina- 
tors of, and resources wielded by, people in collective action (Stevenson and Dryzek 
2014) but does not capture everyday informal talk, and uses methods that are less 
inductive than might appear (Parkinson et al. 2020). 

Big data techniques allow researchers to reconcile the breadth/depth trade-off to 
some extent. While they have been used to analyze deliberative qualities within a par- 
ticular forum (Hudson 2018), they can be used to analyze everyday, public talk about 
recognizably political issues (Mansbridge 1999; Neblo 2015) across multiple venues 
on a scale well beyond the ability of other techniques to manage—into the millions of 
individual communicative acts across thousands of forums—while at the same time 
reducing the amount of deductive researcher intervention. The results are patterns of 
communication that are not immediately obvious to human analysts. The tools are 
not a panacea: they require careful planning, still-rare technical expertise, and, while 
some tools are more inductive than most of the alternatives, still entail researcher 
choices and interpretation at every step. But they open up the possibility of ‘reading’ 
dispersed public conversations in ways that are simply impossible without them. 

Turning the broad, normative, deliberative systems criteria into a set of evalua- 
tive and empirical cues is not a straightforward task. We followed guidelines set out 
in Bachtiger and Parkinson (2019, 133-137) for assessing the deliberative quality of 
a system in a ‘summative, dispersed, emergent sense, as opposed to the ‘additive’ 
approach which looks for moments of deliberation more or less narrowly defined. 
Two issues stand out. 
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First, the ideal unit of analysis is ‘memes; understood as units of meaning that are 
larger than individual words but are much more fine-grained than ‘discourse.’ This 
must not be confused with ‘memes’ as colloquially used today, which refer to often 
humorous images and text spread online. Instead, we refer to memes as clusters of 
words that regularly appear together on a given issue in a given context, relative to 
the ‘norms of appropriateness’ of that setting (John 1999). As will be seen shortly, 
our methods identify clusters of words that approximate the memetic ideal, running 
from 50 to 70 clusters in our cases. 

Second, we focus on the deliberative inclusion criterion rather than reason- 
giving—we are looking at the degree of sharing of perspectives ‘in the wild, which 
is much broader than the requirement that those perspectives and claims be weighed 
and connected with reasons. We do not think that this thematic inclusion criterion 
is all there is to deliberative quality. But we do think it is a necessary condition, and 
big data methods give us a unique way of coming to grips with it. 


Data Gathering 


Our case selection started by conducting a simple online search for use of the 
‘national conversation’ label, something we had already come across in Scotland, but 
which we found in widespread use in the United Kingdom, United States, and Aus- 
tralia. Preliminary archival research and informal conversations with actors involved 
soon led to us focusing on two contrasting cases: a rich, detailed, multi-venue, 
grassroots conversation in the Scottish independence debate, but a relatively thin 
conversation on Indigenous recognition in Australia, in which symbol dominated 
substance. 

Next, we listed a wide variety of potential data sources in the public sphere: 
social media; traditional broadcast and print media; official government information 
sources, including Hansard but also press releases, policy papers, party communica- 
tions, and speeches; blogs including small independent multi-authored and edited 
outlets like Bella Caledonia and Indigenous X; and public forums like Reddit, as well as 
forums that were not obviously political but nonetheless featured recognizably polit- 
ical discussions. In the UK, the latter was easy: the parenting network Mumsnet and 
the entertainment news site Digital Spy both featured threads on Scottish indepen- 
dence that lasted for the entire duration of the debate.’ Australia lacks such forums: 
Facebook dominates social media in Australia to the extent that member discussion 
forums tend to focus on specialist topics and attract relatively few participants (see 
https://gs.statcounter.com). 


 Coined by Dawkins (1976) to refer to units of information in evolutionary processes, the concept of 
a ‘meme is a staple of linguistics and semiotics, but has passed into everyday language as combinations of 
images and words that are playfully adapted to contexts and issues. 

° For discussions of Mumsnet as a public space, see Gamble (2010) and Pedersen (2020). 
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The next step is gathering the data. To do this, one creates small, autonomous 
computer programs called ‘crawlers’ which look for and download, or ‘scrape’, tex- 
tual data from the World Wide Web, using keywords and time, location, and other 
parameters chosen by the researchers. We created our tools in the R statistical com- 
puting environment (Munzert et al. 2014); many others use Python, and there are 
alternatives. But different websites have different rules—Application Programming 
Interfaces (APIs)—which control access, and one often has to contact sources for 
permission, and pay to access their sites in this way, not to mention data protec- 
tion laws that limit the extent to which individuals’ opinions and contributions can 
be collected and reproduced. Most research that relies on web scraping is, therefore, 
conducted on a single platform because it is too time-consuming and expensive to 
have to go to many different platforms and ask permission. We avoided most of the 
cost and some of the API complications by only scraping data from public posts on 
publicly available sources, guided by our ethical protocol and relevant data protection 
legislation. 

Because our research focused on the inclusion criterion, we also needed to check 
whether different perspectives were equally likely to be present in the data. For 
example, in the Scottish case it was much easier to get information from pro- 
independence sources than pro-Union. We were given numerous explanations for 
this, but controlling potential Yes bias in the data was one reason for including 
Mumsnet and Digital Spy: everyday citizens’ fears and concerns were expressed more 
readily in the relatively safe spaces of members’ forums, much less readily in the 
‘hot’ and expressly political public space of Reddit and Facebook campaign pages 
(Wojcieszak and Mutz 2009). In the Australian case, the main issue was the sheer 
fact that Aboriginal and Torres Strait Islander peoples make up 3.3 per cent of the 
population, which could have meant the near-complete submersion of Indigenous 
perspectives in the database. Our solution was to over-sample Indigenous sources, 
leading to a database that was 22 per cent Indigenous, defined as being clearly 
owned, controlled, and/or written by people who identified and were recognized as 
Indigenous.* 

The final consideration is the time period. Such debates do not have uncomplicated 
start and end points. In the Scottish case we had a ‘focusing event, the referendum 
on 18 September 2014, while prior qualitative research and a simple ‘frequency of 
mentions’ count led us to think that the ‘real’ debate only started with agreement in 
October 2012 on legislation allowing the referendum to go ahead. So, we chose our 
starting point as the month before, giving us two full years of discussion before the 
referendum. We set our end point three months after, which captures the report of a 
commission set up to recommend further devolution of powers to Scotland following 
the referendum. 


* The research on First Nations communication was guided by a group of Indigenous activists, aca- 
demics, public servants, and elders. We thank them for their support and guidance. More detail is on the 
project blog at http://natconvblog.wordpress.com. 


Nuria Franco-Guillén et al. 237 


In the Australian case, we were tracking a debate live instead of retrospectively, 
which came with both advantages and disadvantages. Live data is often easier to 
access with all its links and other metadata in place, whereas archiving often strips 
out that detail; some sites’ APIs are less restrictive live, for others the reverse is true; 
and we were more at the mercy of events. There was no agreed date for a referen- 
dum and no question; and there were two separate processes set up under different 
governments, one to generate general awareness and support for the principle of 
recognition, the other to consult widely and recommend a proposal to be put to a ref- 
erendum. The eventual proposal was encapsulated in the Uluru Statement from the 
Heart, released on 26 May 2017,” and this became the focal point of the Australian 
database. 

There is much else that could be said about sources and data gathering, but the 
key points are these: online research requires access to websites that are designed 
and built in different ways, with different levels of public access, sometimes in differ- 
ent regulatory regimes, cultures, and power relations, all of which require different 
access strategies. Using a single data source removes a lot of the complexity but, for 
deliberative systems research, leaves aside key questions about the spread of themes 
around the public sphere, and may erase important differences in respect and stand- 
ing between dominant and subaltern groups. Furthermore, there are choices to be 
made about where to put one’s start and finish lines in the ever-flowing stream of 
public conversation; and whether to follow live debates, or work with historical data, 
or both. 


Tool Selection and Data Preparation 


To be usable, raw data needs to be prepared—‘cleaned’—to eliminate a number of 
known sources of error (Lucas et al. 2017). However, it is also the case that questions 
of gathering and preparation are bound up with questions of how to analyse the data: 
different analytic tools require different preparation approaches. So, we start with the 
tool selection issues.‘ 

Our tool selection was driven by three main considerations: (1) the scale, multiple- 
origin, and heterogeneous nature of the data; (2) our desire to stay as inductive and 
context-sensitive as possible to avoid imposing our thematic preconceptions, remain- 
ing as true as possible to the memetic ideal; and (3) availability of training, advice, 
and support. The first two considerations meant that we rejected simple ‘word count- 
ing’ methods, like content or sentiment analysis, which strip out most context; and 
what Grimmer and Stewart (2013, 273-280) call deductive ‘categorization meth- 
ods, using predetermined dictionaries or supervised machine learning. Instead, we 


5 See https://ulurustatement.org/ (accessed 23 February 2021). 
é The project's ethical protocol and further details of the data search, cleaning, and analysis work are 
available from https://natconvblog.wordpress.com. 
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looked at more inductive, automated ‘clustering’ methods. All three criteria were 
met by Structural Topic Modelling (STM), a package of the R statistical computing 
environment, which applies probability tests for the discovery of topics within a cor- 
pus of text.” STM sorts words into topics, defined by groups of words that tend to 
co-occur according to four different probability tests.® 

Topic modelling has two major advantages over other techniques of textual analy- 
sis: it quickly handles very large corpora; and it does not apply predetermined coding 
or selection frames and thus uncovers ‘latent topics, themes in the corpus that are 
not immediately obvious to human readers. And while it treats documents as ‘bags 
of words, regardless of syntax or context, it maintains the links between a given 
topic and the original source documents that are most likely to feature that topic. 
So, researchers can always drill dowr to the original source data with its contextual 
features still in place. 

One non-inductive feature of STM is that researchers need to make a judgement 
about the number of word clusters or ‘topics’ that the software reveals before running 
the analysis—in STM speak this is called a ‘topic model. One approach is to find a 
model that balances two traits, called exclusivity and semantic coherence (Rothschild 
et al. 2019), but the heterogeneity of our data produced almost identical trade-offs 
for every model; so we could not use this method. Instead, we had to use researcher 
judgement based on prior research and topic correlations: we ran a number of models 
and compared them to find one for each case that was fine-grained enough to disag- 
gregate themes that we knew from interviews had different meanings in context, but 
not so fine-grained that everything correlated with everything else. 

Model selection was an iterative process that went hand in hand with data prepara- 
tion. First, we filtered out usernames and ‘stop words’ (mainly common prepositions 
and conjunctions) that would generate irrelevant or meaningless topics. Second, the 
sheer scale of Facebook data, and forum users’ tendency to cut and paste sections of 
others’ comments before responding to them, could have obscured minority perspec- 
tives, including sources like parliamentary debates. We dealt with this by eliminating 
duplicate posts, aggregating text by the same author, and limiting the Facebook data 
to ‘interactions, defined as a single post with at least one response, of at least 100 
characters in length. Eventually, we were able to obtain workable models with clear 
topics and graphical representations: seventy topics in the Scottish case, sixty in the 
Australian. 


7 We thank Michael Jensen for introducing us to STM. See https://www.structuraltopicmodel.com for 
extensive resources on the method and its application, including a list of publications that use the method. 
Like other topic modelling tools, STM uses Bayesian probabilistic modelling to derive its word associations 
(Lucas et al. 2017; Roberts et al. 2014). The data gathering, cleaning, and analysis was done in the R Studio 
environment (www.rstudio.com), aided by a web-based data visualization tool called Stminsights, https:// 
cran.r-project.org/web/packages/stminsights/vignettes/intro.html. 

ê Compare Hudson (2018) who uses a closely related tool, Latent Dirichlet Modelling, but on smaller, 
homogenous data sources. 
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Results and Their Representation 


STM reveals its descriptive power through results and their graphic representation, 
revealing surprising patterns that go beyond the mere confirmation or rejection of 
prior hypotheses. The main output of a topic model is a table of word correlations, 
the words grouped into ‘topics’ which are defined by four probability tests: 


Probability: the highest probability words associated with a topic. 

FREX: words weighted by their overall frequency and exclusivity to the topic. 
Lift: a weighting that divides probability by frequency in other topics. 

Score: the log frequency of the word in the topic divided by the log frequency 
of the word in other topics. 


PP Oe: oe 


For reasons of space, we have not included the topic tables here, but they can be 
found on the projects blog at http://natconvblog.wordpress.com—the tables include 
short extracts of original text to illustrate. Figure 16.1 below shows the overall topic 
proportions for the Scottish case—again, the Australian figures can be found online. 

The topic proportions give an overview of what is being discussed. The STM soft- 
ware randomly assigns topic numbers (square brackets in the figures) and lets the 
topic words themselves do the talking; but that can be frustrating for visual repre- 
sentation and writing up the analysis, so the topic labels were created by the research 
team as a convenient shorthand. However, it is important not to over-rely on those 
labels, but to refer back to the word clusters and, in most cases, the source texts that 
are most associated with a given topic. 

As well as showing the overall topic proportions, one can also examine topics 
correlations, and Figure 16.2 is a visual representation of those correlations in the 
Scottish model. 

Bringing together the topics based on probabilistic word associations, their pro- 
portions in the databases, and their correlations, starts to generate insight into 
the nature of mass public communication on constitutional matters. We illustrate 
this with what is perhaps the discovery of the research for deliberative democracy 
scholars: the importance of the ‘meta-conversatiom 


The Meta-conversation 


The meta-conversation concerns the degree to which citizens held each other to 
account for upholding norms of democratic debate: particularly giving reasons 
and evidence, calling out abuse, offering tokens of understanding or listening, and 
demanding respect. An alternative label is meta-deliberation (Stevenson and Dryzek 
2014), but we examine features of a well-functioning deliberative system that are not 
necessarily deliberation per se, and use the ‘meta-conversation label to keep that 
distinction clear. 
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Figure 16.1 Topic Proportions, Scotland 


Note: The x axis scale is decimal: 0.075 = 7.5% of the database. On the y axis, the number in 
square brackets is the machine-assigned topic number, for ease of reference to the online topic 
tables. Meta-conversation topics are marked with an asterisk. 


What is striking is how important this is across the three models, and how much it 
varies. In the Scottish model, the most frequent topic, comprising 7.6 per cent of 
the sample, is something we labelled “Hopes & Fears 1’ It is a very general topic 
that focuses on hopes and fears for which side will win, the prospects of changing 
others’ minds, and complaints about fearmongering. It is strongly correlated with 
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Figure 16.2 Topic Correlations Plot, Scotland 


Note: The width of the line represents the strength of the correlation where strong 
is c>0.15; moderate, 0.1<c<0.15; and weak 0.05<c<0.1. Any correlation below 0.05 is 
excluded. 


topic 62 (c = 0.15), about the quality of debate and discussion between fellow citizens; 
strongly (c = 0.13) correlated with topic 47 about persuasion and arguments for 
self-government; and moderately correlated with topic 4 containing allegations and 
assertions about truthfulness and lies. 

The Australian meta-conversation had a different, more polarized tone. The 
biggest topic, at 6.3 per cent of the database, is one we labelled ‘Facts & Rubbish 
and while it does include appeals to the ‘facts’ or requests for evidence, it largely fea- 
tures polarized shouting and personal attacks. But it was not all mud-slinging: the 
strongly correlated “Talk & Listen’ topic featured invitations to listen and expressions 
of gratitude—it is kinder, gentler, with longer contributions, and appears on both 
Indigenous and non-Indigenous sites. Even a topic like “Black & White, which fea- 
tured strongly worded accusations of racism, nonetheless tended to specify exactly 
how someone had offended rather than just slinging the accusation. 

The overall finding is that ordinary citizens can and often do hold each other 
to deliberative standards of argument, evidence, listening, and respect in relatively 
unmoderated, mass public conversation, and not just in the protected, designed 
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spaces of deliberative mini-publics, albeit with differences depending on context. We 
did not expect to find this. We did not go looking for it and would not have found it 
if we had started with a method that relied on pre-analytic coding or normative data 
selection frames. STM allowed us to cast a wide empirical net and then make sense 
of what we caught. 


Data Moderators: Mapping Topics across Cleavages and 
over Time 


So far, we have presented static views of the databases over twenty-seven months. 
STM also allows researchers to show the dynamics of topics, their rise and fall over 
time. One can also code data in STM and show how different variables rise and fall, 
but one does so after data collection and not before, creating data ‘moderators’ either 
by using the metadata of the source, or manually, which is what we did. 

In Scotland, the sources were grouped into ‘Yes, pro-independence sites and ‘No, 
anti-independence ones, only if they clearly declared themselves to be in one camp 
or the other, and generally this was straightforward. We created a neutral cate- 
gory covering everyone else, on the view that allegedly neutral sites would reveal 
their leanings through the topic associations. This turned out not to be the case in 
Scotland—contrary to expectations, we found that Yes and No were largely having the 
same conversation on the same topics. What mattered more was the formal/informal 
divide. We operationalized this as the degree to which the content of a site is pro- 
duced largely by citizens themselves in blogs, user forums, and social media; or via 
political mediators in government, the traditional media, academia and think tanks, 
and the websites of the two official campaigning organizations. We found a clear ten- 
dency for the formal conversation to portray independence purely in terms of Gross 
Domestic Product and similar indicators, whereas the grassroots conversation was 
about whether an independent Scotland was more or less likely to be a kinder, more 
egalitarian place once welfare and health policy was no longer set in Westminster. 
Figure 16.3 shows this formal/informal divide on a topic about the impacts of post- 
Global Financial Crisis austerity measures and social class, which rose in importance 
in informal discussions as the referendum neared but which largely fell off the formal 
radar. 

In the Australian case, we produced a separate model of fifty topics for Indigenous- 
owned sites only, in order to test whether the word clusters that Indigenous peoples 
used were included in the general conversation. If one just looks at the topic labels, 
one can see that there was a broad kind of thematic inclusion going on with only three, 
relatively minor issues not covered at all. However, we found striking differences at 
the memetic level, particularly concerning the everyday experience of racism and 
colonization, and common representations of Aboriginal and Torres Strait Islander 
peoples. For example, we compared topics in each model that relate to the removal of 
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Figure 16.3 Scotland Topic 33, Poverty, Austerity, and Class, by Formality over 
Time 


Indigenous children from their families, part of assimilation policies in the 1950s and 
1960s, and child protection policies now (see Figures 16.4a and 16.4b). Both the gen- 
eral and the Indigenous models mention these, but the former is largely concerned 
with historical child removals, the latter with present ones as well. The same fea- 
ture appears in other topics: non-Indigenous people were more likely to argue about 
whether Australia should ‘acknowledge history’ or ‘move on’; Indigenous people say 
that colonization happens now, and that the strong settler tendency to place these 
things in the past denies reality and responsibility. Conservative groups accuse First 
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Nations of ‘living in the past, but on these issues it is they who focus on history and 
use the past tense. 

Of course, one could reveal broad patterns using discourse analysis, and one could 
identify the past/present tendencies by conducting qualitative textual analysis. And 
we still needed to understand the context to get at what the data was telling us—the 
data did not speak for itself. What big data tools produced was relatively fine-grained, 
inductive detail at scale, circumventing the breadth/depth trade-off. 


Conclusion 


Big data techniques like STM allow deliberative systems researchers to extract qual- 
itative detail at scale, while retaining direct access to the source data, language 
produced by real actors in real time. This allows them to track ‘kitchen table’ ideas 
over time and across major cleavages. There are many choices to make at every stage, 
so trade-offs and researcher intervention do not go away, and other research is still 
necessary to get at what the data means in many cases; but the current state of the 
art allows us to get some insight into everyday talk at a scale that has been simply 
impossible before. This is also a rapidly evolving field, in a rapidly evolving regula- 
tory and commercial context. Yet, even at the beginning of this revolution, we have 
generated concrete evidence of people holding each other to deliberative standards 
in the wild public sphere, and revealed important insights about the nature of the dis- 
cussion and cleavages in such debates that suggest that the summative—distributed, 
emergent—view of deliberative systems is not as far-fetched as it might appear (Owen 
and Smith 2015). New developments will open up more possibilities for thinking 
about deliberative and democratic norms and practices at scale. 
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17 
Qualitative Comparative Analysis 


Matt Ryan 


Efforts to bring deliberative democracy to life are sometimes successful, but not 
always. What accounts for successes and failures in deliberative practice? How can 
we know what works so that we can better design deliberation? How can we know 
what contexts are conducive to deliberation? 

To answer these questions, we need to compare different cases. But if we are 
comparing many cases and at the same time taking their complexity and quality 
seriously, analysing all that information at once is a challenge. Qualitative Compar- 
ative Analysis (QCA) provides an approach and some tools to help us navigate that 
challenge. 

QCA is a method for investigating multiple cases through logical and sequen- 
tial comparison of various explanatory factors. In undertaking QCA, researchers 
establish which groups of conditions always (or almost always) occur (or do not) 
in accordance with whatever is to be explained. The QCA procedure helps establish 
which conditions are necessary to include in explanations. The logical approach to 
comparison helps researchers avoid including superfluous factors in explanations, 
and helps researchers discover which alternative combinations of conditions are 
sufficient for the outcome they wish to explain. 

The QCA approach is considered qualitative because it takes the problem of con- 
structing conceptual boundaries in social research seriously. It is often characterized 
by labour-intensive iteration between evidence and theory. The method requires 
transparency in recording and communicating the researcher’s interpretation of what 
is at stake in measuring social phenomena. This is useful when working with emerg- 
ing or contested concepts. Deliberative critiques of democracy emerged to challenge 
established conceptions of what democracy is. Established ways of observing and 
measuring democracy may therefore not be appropriate for recording and measur- 
ing deliberative democracy. Nevertheless, researchers need to establish valid ways of 
recording what happens in the world that can be easily communicated and open to 
scrutiny by others. 


QCA and Deliberative Democracy 


QCA can help us understand the conditions for good and bad deliberation. QCA 
extends the evidence base for logical comparison across a larger number of cases. 
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Often, the scholarship on deliberative democracy uses research designs which 
typically focus on one or two cases. QCA is a method which can allow for comparison 
across a larger number of cases. QCA maps the constellations of attributes witnessed 
in a phenomenon like democratic deliberation (and the absence of democratic delib- 
eration) to identify what is known and what remains unknown. For example, we 
might want to explain why some governments choose deliberative approaches to 
confronting climate change and others take non-deliberative, for example, techno- 
cratic, autocratic, or laissez-faire approaches. We might hypothesize that alternate 
combinations of different factors explain political choices: say, the presence/absence 
of democratic commitments amongst leaders; agreement amongst advisory bodies 
about an immediate and significant threat to life; strong public opinion for/against 
climate mitigation and organized mobilization for action amongst influential social 
movements. There are eight possible combinations of the presence or absence of each 
of these three conditions that can be represented in a ‘truth table (see Table 17.1). 
Different combinations of factors may act in different directions. For example, where 
the elite actors know themselves to be at odds with mass public opinion, they may 
veto engagement (rows 4 and 5 in Table 17.1). Yet in another scenario where these 
broad groups hold slightly different combinations of commitments or resources, it 
might consistently create the conditions where deliberative strategies are preferred 
(rows 1, 2, and 3 in Table 17.1). The comparative approach will search for informa- 
tion from cases in the real world and ask how consistently deliberation is the political 
choice pursued for each of these possible logical combinations in turn. We might 
find that not all our cases with the same combination of conditions agree on whether 
they use deliberative politics or not, and in those contexts, evidence is contradictory 


Table 17.1 Hypothetical Simple Truth Table: Comparing Conditions for Deliberative 
Approaches to Climate Policy 


Majority 
Advisers of public 
agree opinion 
on/declare and civil Number of 
immediate society cases observed 
Leadership threat to organizers Approach in the study 
Row committed life/climate support to climate with this 
# todemocracy emergency mitigation policy combination 
1 Yes Yes Yes Deliberative 3 
2 Yes No Yes Deliberative 3 
3 No Yes Yes Deliberative 2 
4 Yes Yes No Non-deliberative 6 
5 No No Yes Non-deliberative 5 
6 No Yes No unknown 0 
7 Yes No No Inconsistent 2 
8 No No No Non-deliberative 10 
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(row 7 in Table 17.1). The truth table will also highlight whether some of those combi- 
nations are less well documented than others in empirical work (we find no examples 
in this hypothetical study of the combination of conditions represented by row 6 of 
Table 17.1). 

To take another example, one of the most important but least understood ques- 
tions in deliberative democracy asks when and how different designs or sequences of 
deliberative forums affect policymaking. Almost everyone thinks that answering this 
question is important, but very few have been able to study it well because it is hard to 
do so. In my work with Thamy Pogrebinschi (Pogrebinschi and Ryan 2018), we offer 
a way to respond to this question using QCA. We analyse National Public Policy 
Conferences (NPPCs) in Brazil to find out what combinations of institutional design 
choices are associated with federal legislators proposing bills and passing laws con- 
gruent with recommendations of the deliberative processes. NPPCs are participatory 
processes that sequence several mini-publics from municipal to state and federal level 
in order to recommend changes in specific areas of policy. Our analysis shows that, 
depending on the combination of a range of factors, such as the institutionalization 
of deliberation in a policy area, or the level of civil society influence in organizing a 
process, Brazilian legislators have set agendas and passed legislation congruent with 
deliberative recommendations. Legislators have responded to events that have mobi- 
lized many participants in several deliberations, and alternatively events that have 
mobilized a smaller number of select groups (e.g. Indigenous peoples) in deliberative 
processes. 

Elsewhere I have used QCA to update the literature on how citizen control of 
decision-making is achieved in participatory budgeting (Ryan and Smith 2012; Ryan 
2021). Previous research had considered conditions, such as significant financial 
capacities and civil society influence, necessary for citizen control of processes. Com- 
parison across thirty cases of participatory budgeting using QCA showed that even 
when these conditions were absent, it is possible to identify alternate paths to citizen 
control of policy. Where political leaders were significantly committed to participa- 
tory politics, and bureaucratic support for participation was also substantial, those 
conditions in combination were sufficient for citizen control. Such findings can give 
scholars and practitioners a greater confidence and justification for implementing 
participatory and deliberative projects in diverse contexts. 


Conducting QCA 


QCA involves five main steps,: (1) formulating combinatorial hypotheses; (2) obtain- 
ing and processing data by calibrating sets representing conditions; (3) analysing 
necessary conditions; (4) generating a truth table to analyse sufficient conditions; 
and (5) Boolean minimization to identify minimal solutions. The method can often 
introduce terminology that may be unfamiliar to some readers. Also, these steps are 
not necessarily undertaken in a linear manner. It is therefore useful to first outline 
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what QCA can be useful for, what kinds of research questions it can help to answer, 
particularly in relation to deliberative democracy, and what its limitations might be. 
Here, I will set out how the data is generated and processed using QCA, before finally 
elaborating on the steps by way of an example. 

QCA emerged as an extension of the ‘comparative method’ in social sciences 
(Ragin 1987). Comparative approaches were classically used in explanations of 
large-scale macro-historical political processes—nation-state formation, or democ- 
ratization and revolutions (see, for example, Skocpol 1979). The comparative method 
came to be distinguished from experimental, statistical, and case-study approaches, 
amongst others. These approaches were often portrayed as part of a hierarchy where 
each alternative method mimics the logic of the experiment and the choice of 
which mimicking method is appropriate is mostly decided by what data is avail- 
able (Lijphart 1971). In this hierarchy, research designs with smaller numbers of 
cases might be thought of as less powerful but often more practical, though this is 
a simplification that has been variously challenged and debated. The comparative 
method came to denote the comparison of a relatively small number of cases, often 
based on the logic outlined in John Stuart Mill’s philosophies of scientific inquiry 
(Mill 1950). 

Charles Ragin’s The Comparative Method (1987) is credited with introducing QCA 
in the social sciences. Ragin extended the comparative method by identifying a math- 
ematical algebra which could be applied to formalize some of its assumptions. He 
revealed a correspondence between the assumptions made by many small-N com- 
parativists (where N stands for number of observations), and the mathematics of 
Boolean algebra and set theory (see Ragin 1987, 2000). Ragin continues a tradition 
of juxtaposing the comparative or case-based (including QCA) methods with statis- 
tical methods. While the differences between the two can be unhelpfully overblown 
at times, it is a useful starting point for understanding what QCA can and cannot 
do. QCA is a method for identifying combinations of conditions associated with the 
presence or absence of an outcome. It identifies necessary and sufficient conditions 
for an outcome of interest. As a research method, QCA is usually contrasted with 
correlational approaches which identify the net-effect of independent variables with 
respect to dependent variables (Thiem et al. 2016). 

To take a simplified example, assume we are interested in finding out something 
about the relationship between facilitation and the quality of deliberation in political 
forums. If we want to know whether increasing the number of facilitators increases 
the average deliberative quality across cases, a regression analysis would be an appro- 
priate method of inquiry. It would help to identify correlations amongst variables 
(potentially controlling for certain factors). However, if we want to know whether 
a certain ratio of facilitators to participants is required for deliberative quality in 
the presence or absence of some other conditions—say, stratified random selec- 
tion and/or long vs. short processes—the latter question lends itself to set-theoretic 
analysis using QCA. The key difference between the two methods rounds on deciding 
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if we are asking questions about whether more of x leads to more or less of y or 
whether x is necessary or sufficient for y. QCA cannot answer the former, but it can 
give good answers to the latter question. 


Methodological and Theoretical Assumptions 


QCA encourages researchers to think about when variations in quantity amount to 
variations in quality. Scholars using QCA need to be clear about what observation 
would demarcate, for example, deliberation from non-deliberation or facilitation 
from non-facilitation. To help understanding, it might be useful to contrast QCA 
with experimental methods. In a classic randomized controlled trial, ideally all 
assumptions should be registered; then tests are carried out; and then implications 
for theory are discussed. The idea is that theory and testing are very clearly separated 
to reduce biases in tests of a hypothesis. QCA scholars are often more comfortable 
with reporting changes in the conceptual boundaries they are working with during 
empirical research and analysis. Almost all research approaches must accommodate 
those practices to some degree. Even more formalized approaches aimed at testing 
highly specified propositions will successively pilot and standardize measures and 
engage in theory-informed investigations of the distributions of data to identify the 
most appropriate ways to test their assumptions. 

While standardization is important, and care needs to be taken with any concep- 
tual adaptation, the QCA method allows for a more regular iteration between theory 
building and testing in the course of research. When we are in the field or working 
with data, we may find that we better understand what deliberation is, or what differ- 
entiates good from bad facilitation than we did before and must adapt our measures 
and theories to an extent as we go along. QCA researchers are comfortable with con- 
structing both their populations of cases and the concepts they are testing as they go. 
On the face of it, it might be supposed that this can lead to the worst excesses of social 
science’s cardinal sins: namely accommodating theories to fit existing biases. This is 
an important consideration. In reality, the QCA process accounts for any changes in 
a way that requires researchers to be transparent about their reflections and choices, 
and QCA provides tools for presenting assumptions openly and concisely. This pro- 
cess of refining some aspects of theory based on empirical information and vice versa 
may happen several times in different ways in the course of QCA. 

It is important to emphasize that this juxtaposition of QCA with other meth- 
ods requires some simplification. Most trials are clear that they involve pilots and 
replications, and all scientific inquiry iterates between theory and data. Neverthe- 
less, QCA introduces unique techniques to facilitate that iterative work. Much of 
the work is centred on the process of calibration of sets. Sets are distinct collec- 
tions of objects (e.g. members of the set of nation-states include Australia and 
Ireland but not London or Florence). Calibration is the name for the scientific 
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process of standardizing an instrument for measuring something (think of how 
each thermometer is calibrated to give a standard measure of temperature). QCA 
researchers transform variables and observations into sets with membership crite- 
ria, distinguishing which cases are members of a set, and not members of a set. 
For example, following the ideal/normative conception of deliberation, we might 
say that cases of deliberation are ones where decision-making is supported by public 
reason-giving following the free and unencumbered exchange of ideas. Cases of non- 
deliberation are those where these conditions are absent. So far so good: that crisp 
dichotomization is no different from what we might expect in a theory that wishes 
to utilize some measure of a qualitative or categorical variable. However, when we go 
out in the field, we might find that deliberation in the empirical world is not so easy 
to discern as it seemed in our heads. How do you know unencumbered decision- 
making when you see it? Our rationalities are differently bounded for a start. What 
counts as a reason or reason-giving? 

We may need to be more precise and adapt our conceptual theories in order to 
categorize cases. What counts as reason-giving continues to trouble scholars of delib- 
erative democracy and is a hallmark of their ongoing engagement with the empirical 
world. QCA provides some tools for systematizing this process of refining both 
theory and measurement. 

Ragin borrows the idea of ‘fuzzy set’ membership from computer science, to show 
that cases can have different degrees of membership in a set. To construct or cali- 
brate a set of deliberation, we would need to specify what distinguishes cases that 
are deliberation from cases that are definitely not deliberation; and then also spec- 
ify what distinguishes the cases in between that are closer to deliberation from those 
cases that are closer to non-deliberation. In fuzzy sets, cases’ membership in a set 
varies between a score of 1 representing full membership, and 0 representing full 
non-membership. An important measure of quality is a ‘crossover point’ of 0.5 mem- 
bership which represents the point of maximum ambiguity about whether a case is 
deliberative or not. While fuzzy sets may draw on quantitative measurements, they 
include at least these three qualitative or conceptual ‘breakpoints’ which distinguish 
the assumed quality of a phenomenon (see Ragin 2000 for an in-depth explanation). 
To take a stylized example, imagine set membership scores in the set ‘deliberative 
forum. We might expect that a case of a well-run mini-public is classified fully in 
the set (score of 1), riots are fully out (score of 0). What about parliaments in liberal 
democracies? We might think that these are mostly but not fully deliberative forums. 
They might end up with a set membership score of 0.9 or 0.8 in the set, depending 
on how the set has been calibrated to reflect descriptions of what deliberation is. The 
researcher will need to carefully calibrate their set such that scores accurately reflect 
observations. Any number of empirical observations might actually suggest that par- 
liaments are more deliberative than mini-publics, or that elected politicians are better 
deliberators than citizens in mini-publics, such that recalibration is appropriate. As 
long as researchers are transparent about how and why they construct their measures, 
and the measures describe well the relationships amongst observations in the world, 
the technique is scientifically robust. 
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The second major assumption of QCA approaches is that they treat cases as config- 
urations of key attributes. In this sense those who undertake QCA are less interested 
in isolating the ‘independent effect of a variable than in understanding which mini- 
mal configurations of explanatory conditions are subsets or supersets of an outcome 
condition. Subsets of supersets are observed when all the conditions in the subset are 
also elements of the larger superset. It is this search for subset-superset relationships 
that tests and reveals necessity and sufficiency (see Schneider and Wagemann 2012 
for an excellent explanation). 

For example, imagine a researcher wants to understand what conditions are suffi- 
cient for good deliberation, and they theorize that some combination of government 
support for a process, professional facilitation that is independent from government, 
and a representative selection of participants, will explain good deliberation. The 
researcher might first calibrate a set of ‘good deliberation’ and seek to explain it 
by understanding its relationship with the sets of ‘independent facilitation, ‘rep- 
resentative selection of participants, and ‘government-commissioned process. The 
researcher will then collect their data and perhaps engage in some of the iteration 
outlined earlier. Observations should vary across cases (it will tell us something 
about our conceptualization being quite open or narrow if they do not). If we find 
across all cases that whenever we observe a combination of representative selec- 
tion in a government-commissioned process, good deliberation is never absent, 
we can then say there is evidence this combination of conditions is sufficient for 
good deliberation. This is visualized in Figure 17.1. Note that the area of overlap 
representing a combination of X1 and X3 is the only combination of the hypothesized 
explanatory factors that is a perfect subset of Y. The outcome (good deliberation) is 


Good 


Deliberation(Y 
Independent 
Facilitation(X2) 


Government 
Commissioned i Representative 
Process Mn » Selection 


(X1) | | (X3) 


Figure 17.1 Venn Diagram 


Note: The diagram is showing that a combination of ‘government 
commissioned processes’ with ‘representative selection’ is a subset of, 
and sufficient for, the outcome ‘good deliberation’. * Denotes 
intersection of conditions. 
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a superset of the combination of conditions. Mathematically we are simply saying 
that set scores in the causal combination [X1*X3] are less than or equal to the poten- 
tial outcome Y across all cases. From this we can make some modest generalizations 
(incorporating our knowledge of the field and cases) about whether we expect good 
deliberation in the presence of representative selection and government support for 
processes. 

We might also find that the other hypothesized condition (independent facilita- 
tion) varies in its presence and absence such that there are examples of it being both 
present and absent across all possible combinations of conditions and outcomes. In 
such a scenario, the explanatory value of independent facilitation for good delibera- 
tion is now in question and we can eliminate it as a determinant of good deliberation 
at least in the context of descriptively representative government-commissioned 
processes. 

That is not to say there are no other combinations of sufficient conditions of which 
independent facilitation may be an important component, but it means that we do 
not have evidence for them yet. There are multiple potential configurations of con- 
ditions that may explain an outcome. The Venn diagram in Figure 17.1 suggests that 
the conditions represented do not cover all of the good deliberation set. QCA tries 
to decipher the context in which conditions are explanatory of others and the con- 
texts in which they are not. Numerous software programs are available which can 
quickly compute and summarize the multiple potential superset-subset relationships 
amongst (combinations of) conditions. 

The distinctions in assumptions between QCA and other methods, both those 
typically considered ‘qualitative’ or ‘quantitative, can be overblown. Sensitivity to 
conceptual concerns and how explanations interact are considered in all good 
research in different ways, and over-problematization of such issues can be debili- 
tating. Nevertheless, QCA can be considered to come from a place that privileges the 
concerns outlined in this section over others to an important extent and focuses on 
understanding relationships of necessity and sufficiency. Potential users should be 
aware of these assumptions. The treatment of cases as complex wholes made up of 
combinations of conditions (which could represent different types of deliberation, for 
example), and theories about what is required for, or what contexts are sufficient to 
produce, deliberation are rife in the literature on deliberative democracy. Therefore, 
QCA’s methodological assumptions should lend help in dealing with many puzzles 
that researchers in the subfield face. 


Data and Analysis 


QCA does not preclude any type of data gathering. The key requirement is that data is 
transformed systematically and transparently into a set. A fuzzy set, as we have seen, 
is a set that allows for different degrees of membership. Researchers could rely on 
qualitative data—for example, by specifying membership scores in fuzzy sets ascrib- 
ing to different verbal statements. Researchers can define what evidence would be 
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required for a case to be more or less in the set and assign case scores accordingly, 
based on the evidence. Users can also rely on quantitative data—for example if we had 
a continuous measure of the money spent on several mini-publics, we could calibrate 
the set of ‘well-resourced mini-public’ by specifying the three breakpoints at which 
(i) a low level of spending comes to mean ‘not well-resourced’ and all variation below 
that point is irrelevant to the concept of resourcing a process well; (ii) the spending 
point at which a mini-public is definitely well-resourced by our standards and any 
further spending is an irrelevant luxury; and (iii) the crossover point at which a case 
is more well-resourced than not. We can then ask a software program such as the 
several QCA packages freely available in R to apply a mathematical transformation 
to array all cases’ gradations in spending within this qualitative framework provided 
by our construction. 

Several applications of QCA have also mixed more qualitative and quantitative evi- 
dence. There is no need for the same calibration approach to be used for all sets in an 
analysis. The key requirement is that the sets and measures are developed around the 
research question and they best represent and explain what the researcher is research- 
ing. Sets can even be systematically combined before analysis. For example, we may 
think two sets we have calibrated—let’s say ‘non-interference of privileged actors’ and 
‘non-domination of speaking time’ —add up to the higher-order set of cases of ‘unen- 
cumbered deliberation, and we could calculate cases’ memberships in the latter set 
by considering it the union or intersection of the first two. These processes of com- 
bining set scores are described in depth in the texts recommended for further reading 
at the end of this chapter. 

Once the researcher has created a data set that contains all the membership 
scores for cases in conditions, they can move on to the ‘analytic moment’ of QCA. 
The research will often tack between several analytic moments and calibrations or 
additions of cases and conditions, reinforcing the constructivist elements of QCA 
approach. The analysis, which is quickly performed by software these days, requires 
the researcher only to interpret the outputs of algorithms applied using software 
packages and to specify what levels of uncertainty they would like to tolerate. An 
example will be useful in explaining this process. 


Example: Using QCA in Deliberative Democracy Research 


Here I discuss how we used QCA to assess the impact of deliberative processes, 
and to explain the conditions that were necessary and sufficient for decision makers 
to respond to the recommendation of deliberative forums (Pogrebinschi and Ryan 
2018). More specifically, we focused on how combinations of five institutional design 
characteristics were associated with legislators’ responses to the deliberative recom- 
mendations of NPPCs during the rule of Brazilian President Lula da Silva. We looked 
at whether these processes had been institutionalized or were new to the policy area; 
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the differences between redistributive policy areas and others; the authority of civil 
society organizations; the number of participants; and the number of forums. 

We compared data from several sources from national statistics to coded records 
of conferences and legislative outputs (on the latter, see Pogrebinschi and Samuels 
2014). We first identified those five influencing conditions that could combine to 
explain two different sorts of policy outputs we observed: whether a number of laws 
were passed congruent with recommendations of a conference, or whether a sig- 
nificant percentage of recommendations of a conference were congruent with bills 
introduced in the federal legislature. We outlined several configurative hypotheses 
to explain those outputs. For instance, we expected to see that where NPPCs had 
occurred several times in a policy area, mass participation of citizens in combina- 
tion with strong roles for civil society groups in organizing processes would give 
legislators the confidence to respond to a large proportion of recommendations 
(hypothesis a). Equally, we thought that deliberative processes taking place in the 
context of redistributive policymaking where such processes are less established 
(opening up new areas to deliberative policymaking) would produce laws when 
participation from citizens and civil society was high (hypothesis b). We also hypoth- 
esized that more streamlined designs with fewer sequenced mini-publics would have 
an influence in the context of policy areas not classified as redistributive, or recogni- 
tion that may favour a more in-depth deliberation on more niche or technical subjects 
(hypothesis c). 

To test these assumptions, we calibrated seven fuzzy sets representing the condi- 
tions of interest and coded each cases membership in each set for thirty-one cases 
where we could find adequate data for each condition. We utilized various qualitative 
and quantitative data sources. 

The next step and first stage of data analysis in QCA is to test for evidence of nec- 
essary conditions for the presence or absence of the policy outputs and outcomes 
we were interested in. That is, we asked if the outcomes were a subset of any of 
our conditions. This was an interesting question because there are some debates 
between those who favour participatory or deliberative models of democracy over 
whether mass participation or more streamlined deliberation will legitimate policy 
decisions. Our hunch was that either might lead to responses depending on con- 
texts. Much like in statistical analyses, QCA software calculates measures of fit to 
allow researchers and readers of research to understand and interpret the consis- 
tency and strength of superset-subset relationships. For example, the consistency 
measure allows researchers to interpret the extent to which evidence from all cases 
supports a superset-subset relationship. Our tests showed that neither mass partic- 
ipation nor a more centralized deliberative design were subsets of policy outcomes, 
but that considering one or the other as substitutes (the union of the sets) was highly 
consistent with the necessity relationship. Neither condition is necessary alone, 
but one or the other is almost always necessary. The example shows how QCA’ 
assumptions can accommodate explanations that allow for alternative and equally 
valuable explanations for an outcome, rather than seeing explanations as competing. 
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Following the standard analysis of necessity, the analysis of sufficiency has two 
basic steps: the construction of a ‘truth table, which helps researchers analyse which 
combinations of conditions are consistent subsets of the outcome, and then a pro- 
cess of logical “Boolean minimization, which employs an algorithm to help remove 
logically redundant elements of any explanation. The truth table is a useful visual 
tool for analysis because it groups cases into types according to which combinations 
of presence or absence of key conditions cases most closely ascribe to. An example 
of one of the truth tables for one of the outcomes from our research is reproduced 
below. 

The table shows that despite the number of permutations for combinations of 
presence and absence of the five influencing conditions (thirty-two possible com- 
binations of presence/absence with five conditions), several cases observe the same 
logical combination of conditions. What is presented is only a partial truth table. 
There are several other rows of potential combinations for which we have no good 
observations in the data that are left out here. 

The truth table reflects a summary of a researcher’s data. The consistency (cons.) 
column, sometimes called the inclusion score, gives us a measure of the extent to 
which the sufficiency superset-subset relationship for the combination of conditions 
represented in each row holds across all cases. Sufficiency consistency is calculated 
by summing the minimum values of cases’ set memberships in a proposed causal 
condition and outcome, and then dividing that figure by the sum of cases’ set mem- 
berships in the causal condition (see Schneider and Wagemann 2012). With fuzzy 
sets, all cases will have partial membership in each row (though in the table cases are 
named in the one row which best describes their features). 

The researcher then will have to decide on a threshold at which they can consider 
a combination sufficient for the outcome. Thresholds often range between 0.75 and 1 
and need to be chosen and justified with reference to the researcher’s understanding 
of the cases and theory. Including some inconsistency effectively means researchers 
are happy with an answer that says a combination of conditions is ‘almost always 
sufficient’ for an outcome. 

The truth table can often throw out some surprising results, revealing unusual 
combinations of conditions of cases, or challenging intuitions about what combi- 
nations ought to be considered sufficient for an outcome. Researchers may need 
to reconsider if they have made calibration/coding errors or are missing important 
explanatory variables, or why some apparently very different cases are logically insep- 
arable according to the data. A process of iteration between earlier parts and later 
parts of the research process described in this chapter often ensues. If researchers are 
transparent in explaining how they reached the models they finally present, this poses 
no problems. What is clear is that these tools allow for researchers to systematize the 
process of theory refinement (both conceptual and explanatory) in interacting with 
their data. 

The final stage of analysis is to remove any redundant factors from explanations 
using Boolean minimizations. This means looking at all the rows we consider suffi- 
cient (and possibly some of the rows for which we do not have good examples of 
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cases) and deciding if we can logically remove one or more conditions from any 
potential explanation of the outcome. The process is as described for the ‘indepen- 
dent facilitation’ example in a previous section. There are two advantages here: one 
is that we can use software to help show us neater parsimonious answers to research 
questions, excluding redundant factors; a second is that we can engage in analysis 
of counterfactual cases. Those rows of potential combinations of conditions with no 
good exemplar cases that are not shown in Table 17.2 but are of the type represented 


Table 17.2 Truth Table 


I R C M D P Cons. Cases 

1 0 1 1 0 1 1.000 Environment 2008 

0 0 1 0 0 1 0.928 Aquaculture and fishing 2003 

0 0 0 0 0 1 0.911 Aquaculture and fishing 
2006, Science technology and 
innovation in health 2004 

0 0 1 1 0 1 0.910 Environment 2003, Environ- 
ment 2005, Public Security 2009, 
Sports 2004 

1 0 1 1 1 1 0.898 Cities 2007 

0 0 0 1 1 1 0.846 Culture 2005 

0 0 1 1 1 0 0.792 Cities 2005 

1 1 1 1 1 0 0.740 Health 2003, Health 2007, Social 
assistance 2005, Social assistance 
2007 

0 0 0 0 1 0 0.719 Sports 2006 

0 1 1 1 1 0 0.688 Public Policies for women 2004 

1 1 1 0 1 0 0.588 Health of Indigenous people 2005 

0 1 0 1 1 0 0.563 Oral health 2004, Promotion of 
racial equality 2005 

1 1 1 1 0 0 0.482 Food and nutritional safety 2007 

1 1 0 0 0 0 0.445 Human rights 2008 

0 1 0 1 0 0 0.390 Worker’s health 2005, Youth 2008 

0 1 1 1 0 0 0.384 Public Policies for women 2007 

0 1 0 0 0 0 0.363 Gays lesbians bisexuals 
transvestites and transsex- 
uals 2008, Professional and 
technology education 2006 

0 1 1 0 0 0 0.280 Rights of elderly 2006, Rights of 


Source: Adapted from Pogrebinschi and Ryan (2018, appendix C). 


people with disabilities 2008, Sol- 
idary economy 2006, Sustainable 
and solidary rural development 
2008 


Note: Table shows consistency (cons.) and outcome coded for outcome policy outputs (P). 
I = institutionalized policy conference, R = redistributive policy, C = civil society control of organization, 
M = mass participation, D = decentralized institutional design. Cases are named after the policy area 


and year of the NPPC. 


260 Qualitative Comparative Analysis 


by row 6 in Table 17.1 are often called logical remainders. Using our theories, we 
can investigate the plausibility of assuming such a combination would co-occur with 
the outcome, and consider including those assumptions in the minimization. In 
this way assumptions about possible cases for which we do not have data are made 
explicit. 

In Pogrebinschi and Ryan (2018) we found several sufficient conjunctions of 
conditions that explained our outcomes, which in some cases confirmed our assump- 
tions, in others added nuance, or identified less obvious explanations that we and 
others had not considered. For example, we found that hypothesis (a) was only con- 
sistent within the scope of our non-redistributive policies. Whereas we found that for 
hypothesis (b) the influence of civil society organizations in organizing deliberation 
was less important than we had thought. When we went back to investigate the cases 
in the light of these findings, such interpretations allowed us to make better sense of 
what was going on in the cases. We found evidence for (c) to be somewhat inconclu- 
sive. QCA provided us with the tools to test and refine our theories in a systematic 
and enlightened way. 


Conclusion 


QCA has provided a considerable methodological advance in the social sciences and 
is a good tool for many tasks. I have shown that QCA has been useful for the study of 
deliberative democracy, but its potential in the area is far from realized. I appreciate 
I may have introduced a little jargon for those new to the method, but I hope that 
this short description is clear enough to whet the appetite for some who may feel it 
sounds like QCA is right for their purposes. 

QCA still has limitations and its proponents have responded to several critiques 
in ways that have improved the method. There are also passionate debates amongst 
QCA methodologists. For example, QCA has only very blunt instruments for dealing 
with changes in conditions over time, although improvements are being worked on. 
QCA can be effectively combined with other methods, of course—for example with 
more formal within-case comparison or ethnography. Schneider and Rohlfing (2013) 
provide details on how best to select cases for follow-up studies. 

Deliberative democracy’s origin as a normative critique within democratic the- 
ory is important because this critical perspective and normative project has guided 
action with several cases of different deliberative democratic practices emerging. Set- 
theoretic statements about what conditions are required and what constellations of 
conditions together produce types of deliberation are common. Different designs and 
contexts have different results, and empirical research is needed to explain what is 
happening in practice and to reflect on those theories. QCA offers a useful approach, 
as well as tools and techniques for refining theories systematically in the light of 
observations. 
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PART III 
EXPLORING DELIBERATION 


18 
Ethnography 


Nicole Curato and Nicole Doerr 


In ethnography, argues Lisa Wedeen, there is ‘never nothing going on’ (Wedeen 
2010, 255). Cancelled community forums, sharp dropout rates in mini-publics, and 
incorrectly completed questionnaires may be considered setbacks to understand- 
ing deliberative politics in action. For an ethnographer, however, these may signify 
everyday politics at work. 

Ethnographers conduct research through ‘close-up, on-the-ground observation’ 
in ‘real time and space’ (Wacquant 2003, 5). They provide ‘thick descriptions’ of 
everyday politics from the perspective of those who experience them (Geertz 1973). 
Ethnography lays bare the microfoundations of politics. It recognizes that democracy 
is not just about institutions and political communication, but also esteem, desires, 
hidden injuries, emotions, and aspirations, which shape the meanings that people 
ascribe to collective decision-making (see Lauren et al. 2007). The challenge is to 
render routinized practices visible as well as moments of triumph and madness that 
constitute democratic life, with a view to offering plausible narratives that can ‘rede- 
fine and undermine, negate and create novel explanations about politics’ (Wedeen 
2010, 260). 

Recent years have seen an uptick in studies using ethnography to make sense 
of deliberative democracy. How participatory budgeting is experienced by disen- 
franchised citizens in Brazil (Baiocchi 2005), the precise ways in which disruptive 
communication addresses inequities in democratic deliberation in Paris, Frankfurt, 
Atlanta, and California (Doerr 2018), the frames people mobilize to challenge sci- 
entific discourse in a mini-public on biobanking in British Columbia (Walmsley 
2009), and the anxieties of public engagement practitioners as deliberation becomes 
a commodity to be bought and sold in neoliberal America (Lee 2014) are just some 
examples of ethnography at work. 

In this chapter, we take a close look at the role of ethnography in the study of 
deliberative practice. We characterize ethnography as both a perspective and a tool- 
box (Lauren et al. 2007, 4). In the first section, we focus on the use of ethnography 
as a perspective in deliberative democratic research. As a perspective, we argue that 
ethnography is a fitting methodology to fulfil critical theory’s task of rendering polit- 
ical power visible, particularly in its subtle and insidious forms. Moreover, we suggest 
that ethnography is responsive to recent developments in deliberative theory, partic- 
ularly the increasing recognition of the role of passions, silences, and performativity 
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in what was originally portrayed as an overly rationalistic view of politics (Rollo 2017; 
Wessler 2018; Curato 2019). 

In the second section, we describe ethnography as a toolbox, which offers a 
range of data-gathering and analytical techniques to make sense of public delib- 
eration. Ethnography poses distinct challenges in respect of data integrity. The 
positionality of the researcher is front and centre in this approach, which raises 
issues about the extent to which the researcher can document observations from 
the field, while carrying one’s pre-existing biases, theoretical assumptions, and 
personality traits. Ethnographic research can also stand in tension with deliber- 
ative democracy’s normative commitments. Can the ethnographic virtue of sus- 
pending judgement, for example, be compatible with deliberative democracy’s 
normative commitments of enabling respect, civility, and public spiritedness? 
We tease out these issues by presenting some concrete examples from research 
practice. 

In the third section, we provide empirical examples to illustrate how ethnographic 
methods help to study the content, context, and outcome of deliberative practices 
based on an interdisciplinary set of approaches in political and social sciences and 
sociolinguistics. Here we focus particularly on the examples of studying ‘inclusion, 
‘equality, and ‘transparency’ in deliberative practices and processes by drawing on 
the perspective and toolbox of ethnography. 

Overall, our goal in this chapter is to provide a critical reflection on the uses of 
ethnographic research in the study of deliberative democracy. Our aim is not to 
extol the virtues of ethnography, but rather to put ethnography on the table as one of 
many approaches that can deepen our understanding of how deliberative democracy 
is experienced by ordinary citizens. 


Ethnography as Perspective 


As a perspective, ethnography holds the constructivist view that politics consists ‘not 
of big structures and prescribed roles but of dynamic, contingent interaction among 
persons, households and small groups’ (Tilly 2007, 409). It documents aspects of 
lived experiences to examine how and why agents think, act, and feel the way they 
do (Wacquant 2003). By immersing themselves in the lives of people under study, 
ethnographers can offer a situated perspective that contributes to scholarly debates 
(Wedeen 2010, 257). 

In the context of deliberative democracy, ethnographers typically ask the broad 
question: how is deliberative democracy experienced by ordinary citizens? Ethnogra- 
phy recognizes that there is no ‘natural’ home for good deliberation—not the courts, 
not parliaments, nor citizen assemblies. Deliberation, following André Bachtiger and 
John Parkinson, is ‘contingent, that is to say, it depends on goals and contexts. It is 
‘performative’ for it mixes various forms of democratic communication to justify 
preferences. It is ‘distributed’ because deliberative norms do not take root in a single 
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location, but are dispersed over space and time (Bachtiger and Parkinson 2019, 15). 
Deliberation can very well unfold in the public and in intimate spheres, as private dis- 
cussions amongst friends and family shape our public reason (Tamura 2014). It can 
take place in both secular and religious settings, as in the case of churches serving a 
foundational role in African Americans’ democratic notions of political discourse 
(see Brown 1994). Ethnography, in other words, keeps an eye on the unexpected 
spaces in which deliberative action can take root. 

Deliberative norms are also the subject of ethnographic research. An ethnographic 
perspective embraces the view that ‘there is no Platonic ideal of good deliberation. 


The ideals of which good deliberation is composed are rightly constantly subjected 
to critical scrutiny, examined for unintended implications, opened to revision, 
revised, and subjected again to further contest and future scrutiny. 

(Bachtiger et al. 2018, 2) 


In the introductory chapter of the Oxford Handbook of Deliberative Democracy, 
the editors demonstrate the changing standards for good deliberation over the past 
half-century. The desirability of consensus is one example. Today, most deliberative 
theorists and practitioners recognize ‘the nuances of the pluralist aspirations and 
dimensions of modern democracies’ (Bachtiger et al. 2018, 4). 

Ethnography is a fitting approach to scrutinize the various ways in which delib- 
erative norms are constructed, performed, contested, renegotiated, and revised in 
relation to an issue or polity under study. Consider, for example, the ethnographic 
research of Caroline Lee (2014) on the micropolitics of deliberative democracy 
revealing the rise of a public engagement industry. By observing a series of con- 
ferences and meetings held for public engagement professionals, Lee characterizes 
how deliberative democracy consultants in the United States carefully craft deliber- 
ative forums as unique and special spaces which, in turn, appeals to the sponsors of 
these forums. While Lee does not mean to be dismissive of deliberative consultants’ 
work, her work draws out the implications of ‘deliberative capitalism’ for delibera- 
tive practice. What her ethnographic work contributes to the scholarly conversation, 
however, is the precise description of the time, labour, and skills invested in mak- 
ing deliberative forums relevant for social transformation. And often, as she also 
observes when spending time with participants in these forums, these efforts result 
in little impact, because the ‘larger political context is hostile to the empowering, 
destabilising, progressive bent of deliberative solutions (Lee 2014, 199-200). 

There are many lessons Lee’s work offers, but what we wish to highlight in this 
chapter is the power of ethnography to connect micropolitical acts, such as the impro- 
visations required to pitch the uniqueness of deliberative forums to sponsors, and 
the macropolitical context, such as a hostile political environment. This, we argue, 
allows deliberative democrats to understand the precise obstructions for institutions 
of representative democracy to move in a deliberative direction. 
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If ethnography imbibes the ethos of constructivism, does this mean that it does not 
accept any a priori ‘truths’ about deliberation? Take the case of Amartya Sen's (2003) 
claim that deliberation is a universal human capacity. One approach is to accept the 
universal validity of this claim, and to critically examine how differently this human 
capacity is expressed in different cultural contexts. The goal is not to romanticize cul- 
tures, but to lay bare the structures of power that account for the durability of existing 
political institutions. Edward LiPuma and Thomas A. Koelble’s (2009) work on South 
Africa, for example, characterizes how consensus-oriented village-level deliberation 
resonates with the country’s pre-colonial roots, and therefore makes deliberative 
democracy a more fitting form of democracy than the Western liberal version. How- 
ever, the authors do not wax lyrical about this, and instead point out that traditional 
leadership structures are also composed of ‘power-hungry patriarchs’ who reappro- 
priate the vocabulary of deliberation as a legitimizing device for rigid hierarchies 
(LiPuma and Koelble 2009, 201). By closely observing the behaviours of chiefs and 
councillors during village-level decision-making, LiPuma and Koelble were able to 
generate an evidence-based critical analysis of the quotidian manifestations of power 
that undermine norms of inclusiveness and reason-giving. 

Another approach is to take a step back and interrogate what precise ‘human capac- 
ity’ deliberation entails. One consequence of deliberative democratic research’s focus 
on ‘what people say’ (e.g. analysing transcripts of deliberative forums, pre- and post- 
deliberation surveys, etc.) is the disregard for analysing ‘what people do. The field’s 
methodological partiality to words has the unintended consequence of downgrading 
non-linguistic forms of speech and presupposes that speech is a universal human 
capacity. Clearly, this is not the case. Deliberative communication is multimodal 
(Curato 2019). It is a visual, creative, embodied, and emotive process. Persons with 
speaking disabilities, or those who refuse to speak up for reasons of trauma or resis- 
tance, can offer justifications for their views, albeit in subtle and not easily detectable 
ways (Rollo 2017). Ethnography as a perspective enables us to respond to the chal- 
lenge posed by Toby Rollo of overcoming the ‘methodological subordination of deeds 
to speech; for this erases the contributions of those who are either unable or unwilling 
to exercise their voice in the public sphere (Rollo 2017, 588). 


Ethnography as Toolbox 


How exactly does ethnography capture aspects of public deliberation? What are the 
tools for data gathering and analysis available for ethnographers? 

Ethnography entails ‘immersion in the place and lives of people under study’ 
(Wedeen 2010, 257). While research methods like experiments aim to neutralize the 
context, ethnography places context at the centre of the study to understand how 
realities can be explained by situational factors (Talpin 2012). Ethnographic methods 
often employ a combination of data-gathering techniques, including: 
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Living in the community being examined; learning a local language or dialect; par- 
ticipating in the daily life of the community through ordinary conversation and 
interaction; observing events (meetings, ceremonies, rituals, elections, protests); 
examining gossip, jokes, and other informal speech-acts for their underlying 
assumptions; recording data in field notes that attempt to produce daily accounts 
of social and political life. 

(de Volo and Schatz 2004, 267) 


All these techniques of data gathering are aimed at researchers attuning themselves 
to the rhythms of their subjects’ lived experience. Research is conducted in every- 
day and seemingly mundane settings to understand ‘ways of doing politics’ ‘as they 
happen and where they happen’ (Baiocchi and Connor 2008, 144) instead of staged 
settings like, for example, in-depth interviews organized according to the researcher's 
time and outside the respondent’s everyday routine (Eliasoph and Lichterman 2003, 
743). This is not to say that interviews cannot be part of ethnographic research. We 
underscore the centrality of researching in everyday settings to ethnography as a 
methodology to map recurrent patterns of interaction from the perspective of those 
who experience the social reality under study. 

How do ethnographers analyse data? There are no hard and fast rules, but there are 
considerations worth bearing in mind. The first has to do with the subject of analysis. 
Ethnographers consider mundane aspects of everyday life as key sources of meaning. 
‘Contradiction, silences, and confusiom are angles for data analysis (Wolford 2006, 
340) for these build the character of the communicative environment and speech 
norms in which deliberation takes place. Ethnographic analysis requires sensitivity to 
the nuances of meaning-making which are often inaccessible without an immersive 
method. 

To achieve this, ethnographers engage in a rigorous process of transforming every- 
day observations into analysable data. This may take the form of writing field notes, 
recording reflections and observations in voice memos, taking and annotating pho- 
tographs, and other memory devices that can chronicle significant ethnographic 
moments. Like transcripts being analysed for, let’s say, a Discourse Quality Index, 
ethnographic data is categorized, interpreted, and connected using themes that 
address one’s research questions. Victoria Sanford (2003) gives a helpful introduction 
for researchers to assume a self-reflective methodological perspective to their writ- 
ing process. She suggests that academic writing constitutes a hegemonic translation 
process that slowly transforms ethnographic ‘data into Western scientific knowl- 
edge. For example, Doerr’s work illustrates the limitations of exploring ethnographic 
data in multilingual group settings where the author had to rely on interpreters for 
Hungarian and Turkish (Doerr 2018). In this way, the recognition of one’s own 
limitations in understanding the ‘research subjects’ and their languages and posi- 
tionalities within global hierarchies of research and education is one of the starting 
points for a reflexive ethics of ethnography (Sanford 2003). 


270 Ethnography 


Analysing ethnographic data also entails managing the tensions between the 
ethnographic spirit of suspending judgement in grounded observations while 
remaining committed to the core aspirations of deliberative theory. Some ethnogra- 
phers prefer to provide thick descriptions of lived realities without engaging in moral 
and political judgements (e.g. Snow and Morrill 1995), while critical ethnographers 
argue that the first responsibility of ethnographic research is to uncover the hidden or 
taken-for-granted practices, structures, and manifestations of power that shape lived 
experience (see Madison 2011). We argue that an ethnography of deliberative democ- 
racy is necessarily a critical ethnography, whereby researchers embrace an ethical 
responsibility to identify structures of power and injustice in the polity under study. 
Robust empirical methodologies are the foundation of enquiry, but these observa- 
tions are oriented towards describing the relationship between ‘what is’ and ‘what 
could be’ (Madison 2011, 5), rather than prescribing ‘what should be’ In this pro- 
cess, the values and ethical commitments of the ethnographer are not presented as 
detached from the empirical analysis; rather, they are placed in the foreground and 
opened up to critical scrutiny. 


Illustrative Examples 


Many studies illustrate how ethnography can help investigate and analyse pub- 
lic deliberation or deliberative group settings focusing on normative princi- 
ples such as inclusivity, equality, and transparency within deliberative processes 
(Mansbridge et al. 2015). These principles are by no means the only important virtues 
in public deliberation, but we focus on each of these examples to demonstrate how 
these abstract virtues can be observed, documented, and analysed in ethnographic 
research. 


Inclusion 


Sociologists and critical discourse analysts have developed an elaborate body of inter- 
disciplinary ethnographic methods to address the theoretical interest in empirical 
conditions that facilitate or impede the inclusivity of deliberative group settings. The 
advantage of these studies is that they provide different entry points in ways to study 
the social norms and context settings as well as participants’ conceptions of democracy 
structuring deliberative interaction (della Porta 2005; della Porta and Rucht 2009; 
Polletta 2002, 2015). 

Inspired by feminist and critical theories, critical discourse analysts have devel- 
oped innovative ethnographic approaches to understand how social context con- 
ditions the dynamics of inclusion/exclusion within public deliberation (Young 
1998; Wodak 1996). Within the broad field of critical discourse analysis (CDA), 
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critical historical discourse analysis (CHA) offers a way to combine ethnographic 
observations of non-verbal interaction with the fine-grained analysis of verbal utter- 
ances within public discourse based on sociolinguistic methods (Wodak 1996). This 
method has been applied to the study of online and offline settings, including a 
critical analysis of the broader institutional and social context and the discourses 
surrounding the immediate context of deliberation (Wodak and Wright 2006). 

In another field, within the sociology of civic participation and social movements, 
sociologists have developed cross-national comparative approaches to studying the 
inclusivity of large-scale grassroots democratic forums (della Porta and Rucht 2009). 
One key difficulty that scholars needed to address in this field is how to compare a 
large-scale grassroots deliberative process as part of the European Social Forum, a 
counter public space organized by global justice activists across Europe (della Porta 
2005). For example, an ethnographic study by Donatella della Porta and Rucht was 
focused on global justice activists engaged in deliberations in several European coun- 
tries and taking place in meetings at the local, national, and European levels (della 
Porta and Rucht 2009). While all the various groups studied had, in principle, agreed 
to work together based on the ideal of discursive, communicative, deliberative, or 
consensus-based democracy, the place-specific norms and practices of consensus 
and decision-making varied considerably between radical, participatory, or even 
representative democratic conceptions (della Porta and Rucht 2009). Moreover, the 
understanding of what constitutes inclusive debates and consensus proposals also 
varied across countries and even within different movement cultures within each 
local assembly (Flesher 2015). Our normative focus on deliberation may restrict 
broader theoretical or methodological use of ethnography for the study of civic dia- 
logue. For example, a normatively driven research agenda on a deliberative search 
for mutual understanding can marginalize questions posed by agonistic theories of 
democracy—though recent work shows the intersection between deliberative and 
radical democratic approaches (Ercan 2017). 

Ethnographers of deliberative processes should make transparent their own stan- 
dards of analysis while considering that these differ from participants’ and organiz- 
ers norms of democracy. This is the main lesson of these studies. It is necessary 
to reflect on ethnographers’ standards when interpreting research findings and the 
broader impact of research in dialogue with stakeholders. 


Equality 


The equality of participants in deliberative processes has received attention across 
a wide range of ethnographic studies in social and political sciences, media, com- 
munication, and cultural studies. In addition to political scientists, sociologists 
and scholars of culture in organizations have relied on ethnographic methods to 
develop empirical approaches to the study of deliberative forms of participation, civic 
engagement, and communication. 


272 Ethnography 


Faced with timely questions of inequality, political sociologists in the United States 
have developed ethnographic case studies to examine the promise of egalitarian 
deliberation and participation (Walker et al. 2015). This work comes out of an inter- 
disciplinary tradition of ethnography inspired by feminist and sociological theorists 
revealing challenges of egalitarian group discussions and egalitarian dialogue, both 
within civil society as well as in institutions (Freeman 1972; Fraser 1990; Mansbridge 
1983; Young 1998; Polletta 2002). 

A first ethnographic approach to inequality is provided by studies in the fields of 
civil society and social movement networks, which were interested in observing and 
understanding power differences within ostensibly egalitarian group settings. This 
work illustrates how to observe multiple types of leadership and obvious or sub- 
tle power asymmetries in direct interaction amongst people in group processes of 
decision-making (della Porta and Mosca 2007). For example, power relationships 
that counter egalitarian ideals of discussion can be based on charisma, formal sta- 
tus hierarchies, informal social relations, including friendship or fellowship, or on 
ideological partisan affiliation (Polletta 2002; della Porta and Mosca 2007). 

Another way to approach equality ethnographically has been developed by cul- 
tural sociologists interested in understanding whether and how social class differ- 
ences impede the potential of egalitarian deliberation, focusing largely on American 
civil society groups (Eliasoph and Lichterman 2003; Leondar-Wright 2014). Betsy 
Leondar-Wright (2014) shows the failure of egalitarian discussion and cooperation 
based on class-cultural differences creating internal divisions within progressive civic 
associations in the United States. Her observations show that, in order to have one’s 
voice count equally during discussions, group members need to be familiar with 
implicit cultural codes. Even though formal norms of facilitation should, in principle, 
make sure that all group members are treated as equals, implicit cultural codes define 
whose voices are heard and whose are not. This resonates with earlier work of the 
limits of ‘free spaces’ in movements (Freeman 1972; Polletta 2002). Members of tra- 
ditionally disadvantaged groups may choose to exit deliberative spaces dominated by 
implicit or explicit cultural codes fitting the intersectional gendered norms of equal- 
ity reflecting the culturally specific style of middle-class, white, androcentric speakers 
(Mansbridge 1983; Leondar- Wright 2014;). More studies show how selective habits of 
hearing (Polletta 2002; Doerr 2012) or group styles (Eliasoph and Lichterman 2003) 
limit equality within deliberation and restrict its transformative potential. 


Transparency 


Transparency is key to democracy, both in large-scale formats of public delibera- 
tion and in digital forms of communication, as well as in the context of access to 
local governance and institutions (Fung 2013). Doerr faced the challenge of studying 
transparency in her ethnographic research comparing deliberative decision-making 
in meetings taking place in three countries and at the European level as part of the 
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European Social Forum process. In order to understand how the perception of trans- 
parency varies amongst different groups of participants in meetings across a broad 
set of cases involving twenty meetings, Doerr used a triangulation of ethnographic 
data combined with interviews and questionnaires distributed to participants and 
facilitators of deliberative processes (Doerr 2012). Her ethnographic observations 
enabled her to clarify an open question she had come across in her interviews and 
questionnaires, where the majority of participants who had attended perceived the 
multilingual European meetings to be more transparent than monolingual national 
meetings in their home countries. Deliberative theorists would have probably pre- 
dicted the opposite finding, given that language barriers obstructed transparency in 
European meetings. 

Part of the explanation to Doerr’s counter-intuitive result was based on ethno- 
graphic observation of discussions and decision-making at European and national 
meetings: Participants who had volunteered for translation at the European meetings 
had become increasingly aware of how both subtle and unsubtle dynamics marginal- 
ized the voices of many for whom they were translating in those meetings. At the same 
time, they were aware of how power asymmetries in the national meetings obstructed 
transparency. They had discovered that volunteer translators had the unique agency 
to disrupt the process without being perceived as being out of order, because their 
official job was to witness and address linguistic miscommunications. 

As volunteer translators motivated the enhancement of transparency at the Euro- 
pean meetings, these critical participants used their translating position as critical 
listeners within multilingual deliberation to make sure that no group could with- 
hold relevant information or seek to dominate the deliberative process. Ethnography, 
complementing the data gained through interviews and surveys with facilitators, 
participants, and translators, gave Doerr the tools with which to understand the 
overlooked and powerful agency of third actors within political translation, includ- 
ing participants in meetings who used their position as volunteer translators to 
perform critical political translations in order to address inequities hindering demo- 
cratic deliberation and to entreat powerful groups to work more inclusively with 
disempowered groups. 


Conclusion 


In this chapter we provided an overview of the use of ethnography in the study 
of deliberative politics. We described ethnography as a perspective and a toolbox 
that orients understanding towards the lived experiences of the process of delibera- 
tion, as well as revealing how the deliberative virtues of inclusiveness, equality, and 
transparency can be empirically observed and analysed. 

Before concluding, we would like to flag some of the limits of this approach. First, 
on the practical level, ethnography is a time-consuming method, one that demands 
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intellectual and emotional investment on the part of the researcher. Many ethical 
issues surround ethnography which we cannot enumerate here, including issues of 
reliability of the ethnographer’s account, the ethics of anonymization and covert 
ethnography (see Pachirat 2018). Sometimes, practical realities of academic timeta- 
bles, publication deadlines, and other concerns set a limit to the extent to which an 
ethnographer can immerse herself in the field and acquire a deeper knowledge of a 
particular society. Second, on a conceptual level, ethnography cannot provide a neat 
answer to ‘what caused what’ questions. While experiments are useful in identifying 
variables that caused differences in behaviour, ethnography cannot isolate factors and 
instead brings out the complexity of a situation which, to some audiences, may seem 
unconvincing and tentative. Nevertheless, through our brief discussion, we hope to 
have encouraged researchers interested in the nuts-and-bolts of everyday life to con- 
sider ethnography not only as an approach to research, but also as a normative guide 
to engaging the political. 
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Rhetorical Criticism 


John Rountree 


Rhetoric has a bad reputation in vernacular political discourse. It connotes mislead- 
ing speech or shameless propaganda, and it is haunted by modifiers like ‘mere’ and 
‘empty. Yet, rhetoric does not have these negative connotations for rhetorical schol- 
ars. Rhetoric, instead, can be defined as public discourse that creates meaning and 
moves communities.* Understood this way, rhetoric is central to public life and delib- 
erative democracy, and it privileges citizens not only as voters but as communicators 
(Asen 2004). Rhetoricians see it as a core feature of liberal arts education to teach 
ethical communication practices within a democratic community. 

Notably, ‘rhetoric’ in rhetorical studies differs markedly from conceptions of 
rhetoric often invoked in deliberative democratic theory. Simone Chambers (2009) 
describes attempts in political theory to ‘rehabilitate rhetoric as a legitimate compo- 
nent of deliberation’ (325). Such efforts have been undertaken by thinkers like Iris 
Young (1996, 2000, 52-80), John Dryzek (2010), John O’Neill (2002), and Danielle 
Allen (2009, 140-160). These theorists view rhetoric through ancient Greek and 
Enlightenment philosophy as a particular type of communication—the persuasive 
appeal—that can be isolated and potentially exiled from deliberation. ‘Rehabilitating’ 
rhetoric, in this case, means defending the role of explicit advocacy in deliberative 
encounters. However, from the perspective of rhetorical scholars, these arguments 
start from a flawed premise: rhetoric is a fundamental aspect of communication and 
cannot be excised from deliberation. Attempts to do so would be futile and potentially 
harmful if they portray some appeals as non-persuasive and objective. Rhetoricians 
have shown over the past forty years that persuasion exists in all forms of com- 
munication, and even the seemingly neutral discourses of science are not immune 
(see Ceccarelli 2001; Gross 1990). This is not to say that all persuasion is inten- 
tional, but rather that any communication involves choices, choices that inevitably 
present a partial view of the world and invite us to think and act in particular ways 
(Burke 1966, 46-52). The question of rhetoric and deliberation, therefore, should 
not be whether rhetoric is a legitimate part of deliberation, but rather how best to 
be reflexive about rhetorical dimensions of deliberation (Hauser and Benoit-Barné 
2002; Roberts-Miller 2005; Zarefsky 2008b). 


1 For better or worse, the term ‘rhetoric is itself contested within the field and has no central, agreed- 
upon definition. Wayne Booth (2009) in The Rhetoric of RHETORIC charts thirteen popular definitions 
that have variously been employed. Many scholars hesitate to even define it for this reason. Readers should 
therefore understand that the definition I offer is tentative and open to contestation. 
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In this chapter, I introduce rhetorical methods for studying democratic deliber- 
ation. Rhetorical criticism is the qualitative, humanistic study of rhetorical artefacts 
and processes to understand how rhetoric functions within the public sphere.” Unlike 
rhetorical theory, which teases out the implications of rhetorical processes in an 
abstract sense, rhetorical critics take deep dives into particular cases to understand 
the richness of communicative interactions. Sixty years ago, this meant studying the 
great speakers and speeches, but now rhetoricians study people and artefacts that 
are much more inclusive of the wide variety of communicative practices that char- 
acterize democratic life. Rhetoricians concern themselves both with the exceptional 
and the quotidian, from the high-flown oratory of presidential address to the seem- 
ingly innocuous discourse around how to tally war deaths (Barnett and Boyle 2016; 
Stuckey 2004). Scholars examine objects as diverse as historical photographs, legisla- 
tive debates, petitions, architecture, protests, museums, digital platforms, scientific 
papers, bureaucratic memos, and more. Rhetorical criticism offers a close look at 
specific communication practices and how they constitute meaning in a democratic 
society. 

In what follows, I explain major theoretical assumptions of rhetorical criticism that 
pertain to the study of democratic deliberation. I then introduce the methods of close 
textual analysis that animate current scholarship. 


Theoretical Assumptions of Rhetorical Scholarship 


To understand rhetorical criticism as a method, one must first appreciate how rhetori- 
cians theorize the relationship between discourse and deliberation. I review two 
assumptions in this section: the rhetorical construction of reality and the central- 
ity of context. These two assumptions emerge out of decades of theoretical debates 
concerning the nature of rhetoric and how it shapes democratic experience. 


Rhetoric Constitutes Reality and Invites Deliberative 
Judgements 


The first major assumption rhetorical critics make is that everything we say, do, or 
know is inevitably filtered through a communicative terrain that contextualizes and 
gives meaning to our experiences. Discourse is tied to what we think and what we are 
able to imagine in public deliberation. Rhetoric empowers and constrains in this way, 
as any given piece of discourse invites audiences to share in its view of the world— 
including its blind spots and misconstruals. 


? Most rhetorical scholars would view theory and criticism as complementary projects—good theory 
emerges out of criticism, and good criticism is informed by theory. James Jasinski (2001, 256) calls this 
‘conceptually oriented criticism. 


280 Rhetorical Criticism 


In the context of democratic deliberation, rhetoricians have shown how symbolic 
choices invite specific trajectories in policy debate. Pamela Conners (2017) explains 
that public policy is ‘fundamentally symbolic’ in that ‘the language that analysts, pol- 
icymakers, and the media use to define problems and name solutions shapes the 
nature and extent of the public’s concern and understanding of social issues’ (423). 
David Zarefsky (2005), for example, highlights strategic definitions and how they 
prime policy discourse on poverty to take some directions and not others. As he 
explains, 


If the phenomenon of a person lacking work is symbolized as indolence, sloth, or 
sinfulness, then stern admonition might be called for but public employment pro- 
grams would seem extremely inappropriate. Conversely, if the same phenomenon 
is seen as a specific manifestation of a maladjusted economy, then moralizing or 
temporizing would seem ineffective if not hypocritical; the situation clearly calls 
for public action. 

(Zarefsky 2005, 3) 


This perspective recognizes that rhetoric is both selective and social. It is selective in 
that terms, definitions, or other forms of representation are choices. They simulta- 
neously reveal and conceal, and they are never innocent. The way we describe the 
poor directly implicates how we deliberate on appropriate policy measures for them. 
Simultaneously, rhetoric is social in that people in deliberation participate in a com- 
mon language that helps define the policy situation. Participants, even political elites, 
do not necessarily see ‘behind the curtain’ in understanding the implicit trajectory 
of their symbolic choices, nor are those choices always theirs. Rhetorical choices may 
have developed over time and no longer seem like choices at all. At the same time, 
available rhetorical choices may reflect power relations, as what is ‘say-able’ at any 
given moment can be dictated by the hegemonic ideologies within a society or culture 
(McGee 1980). 

We should distinguish this theoretical perspective from a study of communica- 
tion effects, as rhetorical criticism is not well-equipped to evaluate effects. A handful 
of studies will draw on evidence of how contemporaneous commentators and other 
audiences responded to discourse, but rhetoricians do not isolate causal variables 
or run experimental studies. In fact, the language of ‘effects’ has caused consider- 
able consternation within rhetorical studies in the past, and some rhetorical critics 
are still too loose with terms like ‘persuasive’ or ‘effective’ (Kiewe and Huock 2015). 
Zarefsky (2008a), in an article comparing social science to rhetorical criticism, notes 
that rhetoricians may be mistaken for making effects claims when, in reality, they 
are making claims about how discourse reflects interpretive frameworks and invites 
particular responses. Zarefsky cautions rhetorical critics to be careful with causal lan- 
guage for this reason (2008a, 637). Of course, in a back-and-forth exchange within 
deliberative events, a critic could say that one piece of discourse generated a response 
from others, but they could not make the types of claims survey researchers could 
about political efficacy, trust, knowledge gains, etc. (Kuyper 2018). 
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Context is Central to Understanding Meaning 
within Deliberation 


The first theoretical assumption leads directly into the second. Given that rhetoric 
constructs our understanding of reality and given that people always enter into delib- 
eration within a rhetorical context, understanding that context is very important 
for rhetorical critics. This is because rhetoricians are concerned with discourse as 
it actually operates within the public sphere. Rhetoricians focus on meaning as it is 
generated within existing political communities and cultures, examining discourse 
with all the texture of its context. 

Rhetoricians ask questions that make context central. Because critics are interested 
in how interlocutors create meaning and interpret their own deliberations, critics 
must reconstruct the political exigencies and constraints that bring deliberations into 
existence. In other words, the rhetorical critic seeks to understand deliberation as the 
deliberative participant would—not as an isolated event but as an intervention into 
a historical and political context. In this vein, Kirt Wilson (2010) describes rhetor- 
ical criticism as a ‘hermeneutic enterprise —or an enterprise that investigates the 
connections between a community's discursive texts and their contexts of interpre- 
tation. Rhetorical criticism, according to Wilson, elucidates sociopolitical contexts 
by analysing specific communicative acts because deliberative discourse presents a 
pragmatic intervention into contexts. 

I will offer an example from my own research. In a recent article, I analysed delib- 
eration over the 2013 immigration bill in the US Senate. The bill represented a major 
compromise between advocates of increased border security and those who wanted 
a pathway to citizenship for undocumented immigrants. As I examined the debate, I 
realized that the fundamental disagreement between Democrats and some Republi- 
cans relied on how they were understanding the history of immigration policy going 
back to 1986 with the passage of the Immigration Reform and Control Act. Each 
side reconstructed alternative policy histories—Democrats talked about a history of 
‘enforcement first, while Republicans referenced ‘promises’ that had been broken 
over the last thirty years. Even though I was more concerned with how gridlock 
is justified in congressional debate, my analysis hinged on understanding the pol- 
icy contexts and debates that preceded the deliberation and how they were being 
differently framed and interpreted (Rountree 2018). 

In sum, critics assume that a strong analysis of rhetoric must put it in con- 
versation with its various contexts. For deliberation scholarship, this means that 
rhetorical critics will situate the deliberative encounters they analyse by showing how 
such encounters interact with contexts that are technological (Benoit-Barné 2007), 
political (Hogan 1986; Steudeman 2014), institutional (Hartnett and Goodale 2008), 
patriarchal (Keremidchieva 2012, 2013), ideological (Cloud 2004), and racial (Bacon 
2003; Wilson 2002). Not all these contexts, of course, can be addressed in a single 
study, but they present the range of options available to the critic in the investigation 
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of deliberative discourse. In the next section, I will explicate in more detail how 
rhetorical critics go about this enterprise through close textual analysis. 


Rhetorical Methods: Close Textual Analysis 
of Democratic Deliberation 


Ironically for my purpose in this chapter, it is not customary in contemporary rhetor- 
ical analyses to have a methods section at all, and this genre standard in rhetorical 
journals and books can leave rhetorical methods opaque to those outside the field. 
As it stands, contemporary rhetoricians are reluctant to reduce rhetorical criticism 
to a specific set of procedures, as they perceive it would inhibit the critic’s ability to 
explore and uncover creative perspectives or arguments. 

Nevertheless, rhetorical scholars do have methods—they are just not often outlined 
explicitly. My aim in this section will be to do some translation work for outside 
audiences, to reconstruct methods of rhetorical analysis with an eye towards their 
application in studies of democratic deliberation. I will focus on four key issues: the 
questions rhetoricians attempt to answer about deliberation; the units of analysis they 
rely on; how they construct their data sets; and how they translate textual evidence 
into arguments. 


Asking Key Rhetorical Questions about Democratic 
Deliberation 


Rhetorical scholars study public discourse to answer a wide variety of historical, 
empirical, philosophical, pedagogical, and ethical questions for deliberative democ- 
racy. I will focus on three main categories of research questions that recur in 
rhetorical criticism of democratic deliberation. This list will not be exhaustive but will 
instead focus on the three most recurrent themes driving contemporary research. 

The most dominant strand of rhetoric and democratic deliberation traces the 
mutual interplay between historical public deliberations and political thought. It asks: 
How do historical deliberations reflect and constitute political thought and policy tra- 
jectories? Multiple book-length rhetorical analyses have examined the evolution of 
public policy and how that evolution was carried through rhetorical transformations 
in the public sphere. In line with the assumption that rhetoric constructs reality, 
these scholars ask how deliberative rhetoric justifies some policy trajectories over 
others, and, more broadly, how deliberative discourse reveals undercurrents in public 
understanding. 

Robert Asens work on welfare debates presents a good example of this type 
of research. Asen (2012) shows how images of the poor in US presidential and 
congressional discourse shifted from the 1960s through the 1990s, as those who were 
once viewed as ‘contract-signees’ were gradually construed as ‘cheats, shirkers, and 
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double-dippers. This discourse shift ultimately justified the US Congress repealing 
the Aid to Families with Dependent Children (AFDC). Similarly, Kirt Wilson (2002) 
analyses the US congressional debates over desegregation during Reconstruction. 
Republican idealists at the time pushed for African American civil rights to ensure 
political equality, but Wilson concludes that moderate Democrats constructed a 
‘rhetoric of place’ that justified local, Southern sovereignty and undermined civil 
rights. Wilson’s purpose, however, is not only to understand how the 1875 desegre- 
gation bill was debated but also to understand how key concepts like ‘race’ ‘equality, 
and ‘place’ were understood and contested during this period in history. Deliberative 
encounters, from a rhetorician’s perspective, provide insights into the constructed 
and contested social reality of the times (Wrage 1947). 

A second strand of rhetorical scholarship investigates problematic argumentative 
patterns that arise and repeat themselves in democratic deliberation. Researchers 
in this area will ask: What argumentative pathologies recur in democratic delibera- 
tion, and how should we address them? Scholars offer descriptive and prescriptive 
arguments. They empirically investigate how problematic arguments develop and 
function to hinder deliberation, and they offer normative solutions for combatting 
such arguments. 

Many such studies focus specifically on how deliberators construe political others 
negatively. Robert Ivie (2005) has drawn particular attention to the problem of politi- 
cal conformity in deliberative discourse. He shows how the rhetoric surrounding the 
‘War on Terror’ not only vilified foreign adversaries but also construed dissenters 
as anti-American. Jeremy Engels (2015) also argues that a problematic politics of 
resentment operates within American democratic discourse, one that ‘encourages 
Americans to direct civic resentment against their fellow citizens and not at social 
structures that benefit the wealthy and powerful’ (26). Other forms of argumenta- 
tion have also been examined. Marie Lund Klujeff (2012), for example, argues that a 
‘provocative style’ of deliberation should not be dismissed wholesale, as it can give 
presence to perspectives that may otherwise be ignored. Leah Ceccarelli (2011) exam- 
ines how political operatives ‘manufacture’ scientific controversies in an effort to 
deceive the public on issues such as climate change, and she offers prescriptions on 
how to defend scientific consensus against such attacks. 

A final major theme of rhetorical research is pedagogical and relies on rhetoric 
as training for citizenship in a deliberative democracy. The question, therefore, 
is: How can we develop democratic deliberation as a model of civic pedagogy? 
Craig Rood (2016) puts the matter plainly: this research agenda proposes ‘a linear 
model—rhetorical education and then civic discourse—which promises that the 
things learned in rhetoric classrooms can be useful later’ (144). There are, these schol- 
ars would argue, deliberative and rhetorical habits that we can teach students to apply 
in the larger public sphere. Rosa Eberly (2002), for example, regards the classroom 
as a ‘protopublic’ space where students can practice and produce critical arguments, 
perhaps even for publication in newspapers or for discussion in public events. Along 
the same lines, J. Michael Hogan (2010) argues that rhetorical education can help 
revitalize American civic culture if it teaches students about the ethics of public 
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speaking, the methods of rhetorical criticism, and the history of public address. 
Each addresses deliberation in different ways—public speaking by accounting for 
the judgement of diverse audiences, rhetorical criticism by examining complex pol- 
icy debates, and public address by understanding the historical conditions that have 
promoted public deliberation. 


Approaching the Units of Analysis as ‘Texts’ 


In order to investigate these research questions about deliberation, rhetorical crit- 
ics analyse ‘texts’ as their primary data. Rhetorical criticism originally emerged out 
of literary criticism, and it carried with it an appreciation for the internal workings 
of communicative artefacts. These ‘texts’ can be almost anything—from back-and- 
forth deliberative discussions, to speeches, to photographs, to architecture, or digital 
infrastructure. For rhetoricians, what is important is that they investigate what and 
how people communicate through the products and processes of their communica- 
tive efforts. The critic examines what texts can reveal, how they frame situations, what 
they highlight and downplay, and how they invite audiences to think and act. Close 
textual analysis is a hermeneutical enterprise that puts the internal workings of texts 
in conversation with their various contexts, the understanding of reality they invite, 
and rhetorical theory. 

Rhetorical criticism, as a type of close textual analysis, is good for investigating the 
‘black box’ of deliberation (Mutz 2008, 530; Stromer-Galley and Muhlberger 2009, 
174). Multiple times in democratic deliberation scholarship, writers have called for 
more studies that analyse the communicative practices of citizens as they deliber- 
ate (Black 2012; Tracy 2011, 5). Karen Tracy (2011), notably, has called attention 
to practices of ‘ordinary democracy’ that encompass the everyday communicative 
acts of citizens participating in democratic institutions. Rhetorical criticism is par- 
ticularly well-suited to investigate ‘ordinary’ democratic communication with close 
attention to the implications of what people say in public meetings. In this respect, 
the practice of rhetorical criticism looks very similar to critical discourse analysis or 
some types of argument analyses (Fairclough 2003; Van Eemeren and Grootendorst 
2016; Wodak and Meyer 2015). They emerge out of different intellectual traditions, 
so the differences amongst these approaches are more historical than essential. Com- 
pared to argumentation scholars, rhetorical critics focus less on formal structures 
of argument and attend to aspects of discourse that some argumentation scholars 
would consider beyond the scope of their research, such as narrative, metaphor, or 
identity formation. Critical discourse analysis is harder to distinguish from rhetori- 
cal criticism and, indeed, some scholars would classify rhetorical criticism as a type 
of critical discourse analysis.’ 


° Again, in my estimation, the primary distinction here is intellectual tradition. Rhetorical critics do 
not usually recognize themselves as CDA scholars or cite CDA work, and methods books on CDA seldom 
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Navigating the Intertextual Terrain and Creating a Data Set 


As with any other empirical method, rhetorical criticism requires the researcher to 
construct a data set for analysis. Because a rhetorician’s data set is public texts, the 
data are usually already available—the question becomes which data to examine. Leg- 
islative debates are common fodder for analysis and are a matter of public record, as 
are political speeches and school board deliberations. Additionally, more rhetoricians 
have begun to draw on fieldwork to get better access to vernacular deliberation rather 
than focusing solely on the discussions of political elites. 

Rhetorical critics typically examine public discourse as it permeates through the 
public sphere. Much like the ‘systemic turr in deliberation studies has emphasized 
the interconnectedness of different sites of deliberative discourse over time and space, 
rhetorical critics trace discourse trajectories across a variety of deliberative contexts 
(Parkinson and Mansbridge 2012). Rhetoricians have not yet engaged with deliber- 
ative systems theory, but I would argue that their approach to deliberation has been 
‘systemic for a long time. 

Nevertheless, it is tricky to develop specific procedures for constructing a data set. 
Deliberation at the level of the public sphere presents a vast intertextual terrain that 
is hypothetically boundless. In a thoughtful reflection on the rhetorical analysis of 
policy debates, Robert Asen (2010) notes the challenges of constructing a data set to 
examine: 


To appreciate fully this diachronic policy episode, | would have to place Franklin 
Delano Roosevelt [FDR] address in dialogue with congressional committee hear- 
ings, [US] House and Senate floor debates, government agency reports, news 
media coverage, public advocacy campaigns, and more. Of course, as rhetorical 
scholars interested in public policy, we have to delimit our studies somehow, and | 
do not mean to suggest that macro-analyses must examine every potentially rele- 
vant utterance. Instead, a macro-level analysis must cast a sufficiently wide net to 
elucidate the diverse perspectives forwarded by multiple authors of policy debate 
and the developments in policy debates over time. 

(139) 


Asen articulates the problem well, but the solution remains elusive. What constitutes 
a ‘sufficiently wide net’? The data set is not only laterally expansive but temporally 
so—one could go back and look at policy deliberations through twenty years. He 
seems to suggest that it is enough to gather more texts until one stops encountering 
new perspectives, but again, exactly what that means is opaque. Examining a data 
set that is too narrowly defined would do injustice to the way that contexts shape the 


mention rhetorical analysis. When CDA books do talk about rhetoric, they refer back to classical rhetoric 
rather than contemporary rhetorical scholarship. 
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meaning of the texts. Conversely, a data set too broad would make rhetorical analysis 
an impossible task and would risk losing the richness of the specific case. 

There are no perfect answers here that will apply universally to each study. Instead, 
we need to conceptualize the type of work that the rhetorical critic does when s/he 
constructs a data set. Any deliberative text is going to exist within what Kirt Wilson 
(2005) calls a ‘discursive field, an intertextual terrain that gives the text meaning 
to a particular community. In other words, interlocutors in a deliberation do not 
encounter the deliberation in a vacuum—they encounter it as one in a series of 
communicative moments, and prior discourse provides a framework in which they 
interpret deliberative exchanges. While we can never discover the full context that 
shapes an individual or community’s experiences, we can at least examine the resid- 
ual public discourse available to us and try to understand prominent themes within 
communities of interpretation. 

Rhetorical critics must start with a central object of analysis and build outward 
from that point to reconstruct the relevant discursive field. For deliberation, rhetori- 
cians usually identify the object of their analysis as the controversy—the relevant 
points of disagreement and tension animating a deliberation (Olson and Goodnight 
1994). J. Michael Hogan and Craig Rood (2015) explain that the rhetorical analy- 
sis of policy deliberation ‘involves identifying recurrent themes and key players over 
time, while zeroing in on important “moments” of clash or on key policy “texts” that 
transform the debate or propel it in new directions’ (365). Rhetorical critics, as Asen 
argues, have to take both micro and macro approaches, where they zoom in on spe- 
cific moments while keeping the larger sociopolitical contexts in view. A critic can 
begin, for example, with a transcript of a specific deliberative meeting and locate cen- 
tral frames (e.g. key terms, tropes, stories) that encapsulate the different perspectives. 
These key frames can then be used as a filter as the critic expands outward to look 
at the discursive field into other relevant texts that help contextualize the delibera- 
tion, including news reports, speeches by important policymakers, widely circulated 
TV ads, viral YouTube videos, iconic photographs, etc. Those other relevant texts are 
typically treated as contextual and supportive rather than primary data, but they can 
lend explanatory depth to frames that arise within a deliberative text. 


Translating Textual Evidence into Empirical Arguments 


Rhetorical analysis encompasses much more than what makes it into a final essay or 
book. The rhetorical critic must dwell in the deliberative texts and contexts, shift- 
ing back and forth between potential arguments and a holistic view of the evidence. 
In this process, the critic looks for presences and absences. What is featured in the 
discourse? Are there specific stories, people, terms, metaphors? Conversely, what is 
missing? What rhetorical choices were not utilized? As a critic asks these questions, 
s/he can discern patterns within the texts and start to piece together how they reflect 
and constitute participants’ self-understanding of the deliberation. 
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Specifically, the critic can translate textual evidence into broader empirical argu- 
ments by looking for coherence, recurrence, and contextual probability. Coherence is 
whether the interpretation forwarded by the critic makes sense in terms of the spe- 
cific interlocutor and other statements they have made throughout the deliberation. 
Recurrence is whether the pattern holds up with other deliberative participants or 
whether it is an isolated case. Contextual probability ascertains whether the empir- 
ical analysis makes sense in terms of the discursive field, in terms of the contextual 
exigencies that give rise to the discourse. With each criterion, the researcher should 
also look for counter-evidence that would complicate or refute the argument. 

For example, in Asen’s (2015) book on school board deliberations, he examined 
deliberations concerning permission for a Gay-Straight Alliance to form at one of 
the schools. He noted how school board members kept referring to their own public 
responsibilities to be ‘neutral’ and ‘unbiased} while also expressing personal feelings 
that might conflict with their public obligations. This textual evidence, Asen argued, 
revealed ideological tensions within the school board. He demonstrates recurrence 
by showing how this language came up with multiple board members across spe- 
cific deliberative texts. At the same time, he dives deeper into each interlocutor’s 
statements, giving ample context on other statements they had made and how those 
statements further evidenced coherence of their ideological tension. Asen also recon- 
structed a deliberative context through previous board meetings and newspaper 
accounts, and this contextual work lends probability to the existence of an ideological 
tension. 


Conclusion 


Overall, rhetorical criticism of deliberation constitutes an iterative and interpre- 
tive process, one that straddles the line between inventional art and social science. 
Rhetorical critics closely analyse public discourse to investigate the way it undergirds 
deliberative judgements and bolster healthy democratic practices. While rhetorical 
criticism often does not dictate standard procedures of verifiability and data col- 
lection, it is still a method that demands accountability to the lived experiences of 
deliberators and the ways that rhetoric configures those experiences. Rhetoricians 
emphasize the interplay of texts and contexts within deliberative discourse, simulta- 
neously revealing how discourse invites public judgement and how such discourse 
should be evaluated. 
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Process Tracing 


Jonathan Pickering 


Proponents of deliberative democracy argue that authentic, inclusive, and conse- 
quential deliberation has intrinsic value; that is, it constitutes what democratic 
legitimacy looks like (Dryzek 2000, 173-174). At the same time, deliberation may 
deliver a range of other empirically observable benefits, including higher-quality 
decisions, political systems that are better at managing collective problems, stronger 
capacity of citizens to engage in political processes, and higher citizen trust in 
political institutions (see generally Chambers 2003; Gutmann and Thompson 2004; 
Dryzek 2010).* Some of these empirical claims—such as the idea that involvement in 
deliberative processes enhances participants’ deliberative capacity—may be readily 
supported by surveying participants before and after their involvement in deliber- 
ative events. Yet even in these cases, it often remains unclear how—that is, through 
what kinds of cognitive, communicative, or material processes—deliberative capacity 
is enhanced. Other empirical claims may be even more difficult to substantiate, par- 
ticularly when it comes to evaluating the impacts of small-scale deliberative forums 
on broader political systems, for example the role of such forums in enhancing the 
quality of public debate. Efforts to identify the macro effects of deliberation face the 
methodological challenge of disentangling causal connections when a much wider 
range of actors and institutions shape political systems. 

Process tracing offers a promising and versatile method for qualitative assess- 
ment of these issues because it can identify and test causal mechanisms through 
which deliberation may produce change at multiple scales, ranging from individ- 
uals changing their minds to political institutions becoming more supportive of 
public participation or becoming better at solving problems. For the purposes of 
this chapter, process tracing is defined as ‘the analysis of evidence on processes, 
sequences, and conjunctures of events within a case for the purpose of either devel- 
oping or testing hypotheses about causal mechanisms that might causally explain the 
case’ (Bennett and Checkel 2015, 7). As elaborated below, this definition emphasizes 
the value of process tracing as a method for causal explanation of small-n case studies. 

Despite its increasingly common usage in other areas of political science, process 
tracing is rarely invoked explicitly in literature on deliberative democracy (Bennett 
and Checkel 2015, 4). Following an overview of the process tracing method that 


1 These empirical claims are variously referred to as epistemic, problem-solving or instrumental benefits 
of deliberative democracy. 
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outlines its rationale, potential applications, and limitations, I identify three areas in 
which process tracing can advance research on deliberation: (i) the adoption and dif- 
fusion of deliberative practices; (ii) the influence of actors on deliberative processes; 
and (iii) the influence of deliberative practices on policymaking and broader political 
systems. 


Process Tracing: A Methodological Overview 
Scope and Objectives of Process Tracing 


Collier (2011, 823) describes process tracing as ‘a fundamental tool of qualitative 
analysis. Although variants of process tracing can be found in the work of historians 
going back many centuries, the term was first used explicitly as a methodologi- 
cal description in the 1960s by cognitive psychologists investigating the processes 
through which individuals make decisions (Bennett and Checkel 2015, 5). It was not 
until the late 1970s that the method was applied to political science (George 1979). 
Process tracing has ontological groundings in positivist theories of causal explana- 
tion, such as scientific realism and pragmatism (Bennett and Checkel 2015, 10, 14). 
As such, it is less clearly compatible with postmodern or poststructuralist concep- 
tions and critiques of causation. It also places stronger emphasis on explanation 
than interpretive studies tend to do (Ercan et al. 2017, 198), although it shares with 
interpretive methods an interest in theoretically informed description and analysis 
of individual cases (see also Hopf 2007).? 

The focus of process tracing on understanding causal mechanisms indicates that 
its primary value is in explanatory analysis, although proponents emphasize that 
good description of processes remains an essential foundation for sound explana- 
tion (Collier 2011, 823). Whereas quantitative methods such as regression analysis 
focus on causal effects—such as the average effect of an independent variable (e.g. 
democratization) on a dependent variable (e.g. government performance) over a 
large number of cases—process tracing focuses on identifying the causal mechanisms 
that link causes and effects within a single case or a small number of cases. Causal 
mechanisms are ‘ultimately unobservable physical, social, or psychological processes 
through which agents with causal capacities operate, but only in specific contexts 
or conditions, to transfer energy, information, or matter to other entities’ (Bennett 
and Checkel 2015, 12). Since causal mechanisms themselves are not directly observ- 
able, researchers need to identify observable implications that should tell us whether 
certain mechanisms are at work. Talk of causal ‘mechanisms’ need not assume a 
deterministic conception of causation (where X cause always produces Y effect) but 


* Although Bennett and Checkel (2015, 15) cite several attempts to bridge these differences (e.g. Hopf 
2007). 

> Pouliot (2015) outlines a variant entitled ‘practice tracing’ that draws on both positivist and interpre- 
tivist approaches. 
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may be compatible with a probabilistic conception (where X increases the likelihood 
of Y: Trampusch and Palier 2016, 442). While careful analysis of causal mechanisms 
can illuminate understanding of how causes produce outcomes, it can also help to 
establish whether outcomes are in fact the result of hypothesized causes (on the 
assumption that if certain causes are at work, they will operate through some causal 
mechanisms but not others). 

One advantage of process tracing compared with regression analysis is its ability 
to deal with causal complexity such as endogeneity, interaction effects, and feedback 
effects (Collier 2011, 824; Vanhala 2017, 94-95). In the context of deliberative democ- 
racy, this feature may help to untangle which amongst many purported elements of 
good deliberation (e.g. civility, reason-giving or accurate information) tend to pro- 
duce desired outcomes (e.g. opinion consistency or social trust: Mutz 2008), given 
that factors such as social trust could in turn influence people’s inclination to give 
reasons or interact respectfully. 

Process tracing can serve three main objectives: to build theories, to test theories, 
and to explain the outcomes of individual cases (Beach and Pedersen 2013; Bennett 
and Checkel 2015, 7-8). First, process tracing for theory building involves generat- 
ing hypotheses from the evidence presented in the case. Theory building may lead to 
the discovery of new causal mechanisms. Second, process tracing for theory testing 
involves examining whether theories about causal mechanisms are borne out in the 
case at hand. Using process tracing in this way offers a means of assessing the mer- 
its of competing theories (e.g. agent- or structure-oriented theories of institutional 
change). Third, process tracing to explain an individual case will often involve a mix 
of theory testing and theory development (Bennett and Checkel 2015, 8). 

An illustration helps to show how process tracing is distinct from, but comple- 
ments, other methods of causal explanation. Hanusch (2018) employs both regres- 
sion analysis and process tracing to investigate the influence of democratic quality 
on how well countries perform in addressing climate change. Hanusch conducts 
a large-N econometric analysis that finds a positive correlation between countries’ 
democratic quality and their climate policy performance. While this is a promising 
finding, it leaves open the question of why this correlation emerges: is it because civil 
society groups have more access to policymaking processes, because governments 
are held accountable for their promises, or for some other reason? Hanusch then uses 
process tracing to identify which mechanisms may have been at work. Process tracing 
helps to test the theory on which the regression analysis is based (i.e. that democratic 
quality has a positive influence on climate policy) and to build a theory as to why this 
correlation occurs (Hanusch 2018, 52). He presents a case study of climate policy 
in Canada that has exhibited above-average democratic quality but very low climate 
policy performance and thus represents an outlier in the regression results and a hard 
or least-likely case for the purposes of case-study selection. By using a longitudinal 
design covering the period 1995 to 2012, Hanusch is able to track variations in demo- 
cratic quality and climate policy performance and isolate key causal mechanisms that 
link democratic quality and climate policy performance. He finds that accountability 
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had a strong positive influence throughout the case-study period, while the inclusion 
of stakeholders in policymaking had a variable effect that depended on the quality of 
participatory processes (Hanusch 2018, 234). 


Strategies for Data Collection and Analysis 


When it comes to the data required for process tracing and the strategies needed 
to collect it, process tracing has much in common with other qualitative methods. 
Because process tracing involves reconstructing processes that have occurred in the 
past, it has close affinities with techniques of historiography (Bennett and Checkel 
2015, 8). Overviews of process tracing generally emphasize the need to draw on mul- 
tiple sources of data about decision-making and other political processes, including 
official documents, media reports, stakeholder interviews, event observation, and 
ethnography (Collier 2011; Bennett and Checkel 2015). Triangulation across multi- 
ple data sources is important for increasing confidence that the hypothesized causal 
mechanism is at work and for reducing the potential for biased results. Depending 
on the types of causal mechanisms in question, various methods for data analysis— 
including text analysis and discourse analysis (Neville and Weinthal 2016)—could 
be used as part of a process tracing strategy, for example to explain how a proposal 
or discourse emerging from a deliberative forum gains momentum through media 
reporting or through uptake in policy documents. 

What arguably sets process tracing apart from other qualitative methods in social 
science is its structured approach to evaluating individual pieces of evidence.* Pro- 
cess tracing involves what Mahoney (2010, 131) describes as an ‘implicit Bayesian 
approach in which singular pieces of data may carry great weight’ depending on how 
strongly the evidence supports or contradicts the researcher’s prior theoretical beliefs. 
Guides on process tracing commonly outline a series of diagnostic tests that can be 
used to assess whether a given piece of evidence supports a hypothesized causal infer- 
ence. Collier (2011, 825, building on Van Evera 1997 and George and Bennett 2005) 
distinguishes four tests involving different combinations of necessary and sufficient 
conditions: 


Straw-in-the-wind: neither necessary nor sufficient to confirm the hypothesis, but 
shows its relevance; 

Smoking-gun: sufficient but not necessary to confirm the hypothesis; passing the 
test weakens rival hypotheses; 


* This again has much in common with historians’ methods for analysing evidence, which has led some 
commentators (e.g. Dowding 2016) to question whether process tracing really is a distinct method. How- 
ever, Bennett and Checkel (2015, 9) suggest that political scientists conducting process tracing tend to 
foreground their methodological steps more explicitly, whereas historians prefer to keep them in the 
background to avoid disrupting the narrative they wish to present. 
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Hoop: necessary but not sufficient to confirm the hypothesis; passing the test 
weakens rival hypotheses; and 

Doubly decisive: both necessary and sufficient to confirm the hypothesis; passing 
the test eliminates rival hypotheses. 


Applying these tests is often likened to detective work and frequently involves coun- 
terfactual analysis to gauge whether the same result could have occurred even if other 
causal mechanisms were at work (Collier 2011, 825). These tests could be applied 
to assess deliberation in various ways. Take, for example, the question of whether a 
deliberative citizens’ forum (or ‘mini-public’) influenced a policy decision. A hoop 
test would be whether the report of the mini-public was available to decision makers 
early enough to inform key stages in the decision-making process (see also Jacobs 
2015, 65). If the report was only finalized after key decisions were made, this would 
eliminate the hypothesis of influence. A straw-in-the-wind test would be whether 
the mini-public was covered in media sources that typically attract decision mak- 
ers’ attention. This may suggest a possible channel for influence but would not rule 
out the possibility that decision makers ignored the media reporting or were influ- 
enced by the mini-public via other channels. A smoking-gun test could involve the 
presence of official documentation declaring that decision makers had decided to 
accept the findings of the mini-public. That would provide strong evidence of influ- 
ence (although there remains the slight possibility that the decision was made for 
other reasons and the mini-public’s findings were used as a fig leaf), but it is possi- 
ble that the mini-public influenced the outcome even if no such public declaration is 
made. Doubly decisive tests are difficult to pass in most real-world contexts involving 
multiple interacting causes, although it may be possible to approximate such a test 
by combining elements of the first three types of tests (Bennett 2010, 211). A test that 
comes close to being doubly decisive could involve establishing that decision makers 
accepted the findings of a mini-public despite evidence that civil society organiza- 
tions, key decision makers, and the broader public previously favoured a different 
result. 


Limitations and Synergies with Other Methods 


Process tracing has attracted several critiques, some of which are also levelled against 
other small-n qualitative case-study methods. These critiques include the difficulty 
in producing findings that can reliably be generalized across a large number of cases 
and the related difficulty in detecting average or probabilistic causal effects (Bennett 
and Checkel 2015, 13-14; Collier 2011, 828). Four responses can be made to these 
critiques. First, as with other qualitative methods, careful case design can address 
some limitations of small-n analysis, for example by choosing most-likely or least- 
likely cases that could provide valuable evidence for or against a theory (Bennett and 
Checkel 2015, 13). Second, by analysing multiple stages in a policy process, process 
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tracing can increase the number of observations on which research findings are 
based, even within a single case. An example of this can be found in Ravazzi’s (2017) 
article (discussed below), where a causal mechanism is tested at two stages in a leg- 
islative process. Importantly, the observations generated from process tracing (often 
referred to as ‘causal process observations’) are qualitatively different from the obser- 
vations routinely used in quantitative analysis (‘data-set observations’). An important 
distinguishing feature is that some causal process observations—for example, those 
that pass a ‘smoking-gun’ test—may have high evidentiary value for testing a the- 
ory even if the overall number of observations is small. Third, process tracing does 
not need to focus solely on a single case but can be incorporated into a compara- 
tive case study (Bennett and Checkel 2015, 29; see also Ulibarri 2015, discussed in 
further detail below). Finally, process tracing can be used in conjunction with other 
qualitative methods such as qualitative comparative analysis (QCA; Beach 2018) or 
with quantitative methods such as regression analysis to encompass a broader range 
of cases (Collier 2011, 825). 

Even if the critiques outlined above can be addressed, an enduring limitation of 
process tracing is that its emphasis on careful sifting and analysis of a wide range 
of data sources makes it a highly time-intensive method. As a result, it may also 
be difficult to present the results in brief form (Trampusch and Palier 2016, 442), 
such as a single journal article, although some scholars have shown how this can 
be achieved (see, for example, Ravazzi 2017; Vanhala and Hestbaek 2016; Allan and 
Hadden 2017). One strategy for presenting process tracing results in journal articles 
is to summarize key findings in the main article and give a detailed description of the 
case in a data appendix (see, for example, Ulibarri 2015). Another limitation—albeit 
one that is again not unique to process tracing—is the difficulty of reconstructing 
decision-making processes when documentation for some key stages is unavailable, 
for example if some meetings were closed to the public or not transcribed (Ulibarri 
2015, 298). 

A final risk associated with process tracing is the temptation to think that a 
detailed sequential description of a process is all that is needed to qualify as pro- 
cess tracing, when what is vitally important is that researchers are rigorous and 
explicit about the theoretical framework they are using and about the explanations 
and causal mechanisms that they are investigating to test or further develop the 
framework. 


Examples of Process Tracing in Deliberation Research 


In this section, I show how process tracing offers a useful method for addressing 
three key areas of research in deliberative democracy: (i) the adoption and diffusion 
of deliberative practices; (ii) the influence of actors on the outcomes of deliberation; 
and (iii) the influence of deliberative practices on policymaking and broader political 
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systems.’ Process tracing in all three areas can help to address one of three key 
questions that Elstub et al. (2016, 140) identify as a priority for future research in 
deliberative democracy: ‘how to conceptualize the connections and transmission 
across different sites of deliberative activity’ (see also Boswell et al. 2016). More 
specifically, the second area speaks to ongoing scholarly interest in the internal 
dynamics of deliberation, including patterns of inclusion, inequality, and power 
(Mansbridge et al. 2010), and the third area addresses another key question high- 
lighted by Elstub et al. (2016, 140): ‘how to understand and study the relationship 
between mini-publics and the broader [deliberative] system’ (see also Goodin and 
Dryzek 2006).° In each area I critically analyse existing studies that have used process 
tracing and identify scope for future applications to deliberative democracy. 


Adoption and Diffusion of Deliberative Practices 


The first area where process tracing could offer valuable insights involves explain- 
ing how and why deliberative practices and innovations are adopted and how they 
diffuse.” This question is of considerable interest for scholarship on deliberative 
democracy because it speaks to the political feasibility of strategies to advance delib- 
erative democracy and may help to identify what barriers may be encountered when 
employing these strategies and how they could be overcome. 

Ravazzi (2017) offers one of the few systematic uses of process tracing in the lit- 
erature on deliberative democracy. Her study seeks to explain legislative reforms 
in Tuscany, Italy, which embedded processes for citizen deliberation in the politi- 
cal system. Ravazzi (2017, 81) uses process tracing ‘to highlight the main variables 
that intervened in the development of the process and reconstruct the mechanism 
through which the policy was initiated: The analysis aims thereby to go beyond gen- 
eral explanations offered in earlier literature for how the reforms took place, such as 
public disaffection with government decisions and the presence of a political leader 
(or ‘policy entrepreneur’) committed to public participation (Ravazzi 2017, 82). The 
law in question was first introduced in 2007, with a sunset clause then legislated on 
a longer-term basis in 2013, offering an opportunity to test whether the same causal 
mechanisms applied at both points in time (Ravazzi 2017, 84). 

Ravazzi finds that, although the legislation was passed at both points, it did so 
despite concerns amongst civil society organizations that deliberative forums could 
bypass their established channels of access to decision makers and bureaucratize 
participatory processes that should ideally emerge from ‘bottom-up’ rather than 


° Another area not discussed here in detail involves tracing the process of individual preference change 
that may result from people’s involvement in deliberative forums. 

é The third question identified by Elstub et al. (2016, 140)—‘which standards to employ when assess- 
ing the deliberative quality of a system as a whole’—mainly involves matters of evaluation rather than 
explanation, so process tracing is likely to be less relevant for that purpose. 

7 On the use of process tracing for studying policy diffusion in general, see Starke (2013). 
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‘top-down regulation. Legislators were concerned that citizen forums could erode 
representative arrangements and prolong already lengthy processes for approving 
government projects (Ravazzi 2017, 89-90). Her interviews reveal that a key factor, 
which ultimately outweighed this opposition, was a perception amongst legislators 
and civil society organizations that public participation was ‘politically correct, and 
that it would be ill-advised to oppose moves to strengthen citizen participation 
because it would exacerbate public disaffection and enable right-wing political parties 
to claim the higher moral ground on commitments to public participation (Ravazzi 
2017, 90, 94). Introducing a sunset clause in the original legislation helped to tip the 
balance towards approval. Ravazzi (2017, 94) finds that in the lead-up to the second 
round of legislation, opposition to deliberative legislation remained considerable, 
but again a sense that opposing public participation had become ‘taboo helped to 
overcome this resistance. 

The analysis shows not only that one cause posited by previous research (e.g. pol- 
icy entrepreneurship) was insufficient to explain the process, but also that another 
posited cause (public disaffection) operated in a more nuanced way than previously 
understood: instead of public disaffection producing a positive consensus amongst 
decision makers and civil society on the value of public participation, it was deci- 
sion makers’ perceptions about the value that citizens placed on public participation 
(and the risk of further disaffection if this value were to be undermined) that served 
as an intervening factor that overcame ongoing opposition to the legislation. In this 
way the paper shows how process tracing can yield a fine-grained account of how 
deliberative practices are adopted. 

Even so, the analysis has some limitations. For example, the study does not estab- 
lish definitively that decision makers’ views about the political correctness of public 
participation rested primarily on normative commitments rather than on mate- 
rial interests, such as electoral competition with populist right-wing parties. Jacobs 
(2015) offers a range of useful recommendations on how process tracing could assess 
the effects of ideas (including social norms) vis-a-vis material interests, including 
investigating whether normative beliefs and material conditions varied over time 
(e.g. in this case, whether decision makers held the same views about public par- 
ticipation even if competition from populist parties was higher at some points than 
others). 


Influence of Actors on Deliberation 


The second area that process tracing could illuminate is the influence that actors exert 
on the outcomes or quality of deliberative processes. This could involve the internal 
influence of participants formally involved in a deliberative process (e.g. members of 
a citizens jury) or the external influence of actors on a process in which they were not 
formally involved (e.g. the role of social movements in changing a policy decision). 
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Process tracing has been used successfully to this end in a number of studies on the 
influence of non-government organizations (NGOs) in global environmental politics 
(Vanhala 2017, 89). Betsill and Corell (2001, 77) offer a useful typology of three stages 
at which evidence needs to be gathered: (a) whether NGOs engaged in an intentional 
effort to transmit information to other actors (e.g. decision makers); (b) whether 
the other actors received this information; and (c) whether changes in the actors’ 
behaviour are consistent with the information that the NGOs provided. 

Process tracing could break these stages of influence into further sub-components, 
including the strategies used by actors to transmit information or exert influence. 
Allan and Hadden (2017) provide a good example of this approach in their explo- 
ration of the framing strategies used by NGOs in UN negotiations on loss and damage 
resulting from climate change. To establish that NGOs’ reframing of the issue of 
loss and damage helped to produce a breakthrough in negotiations, they ascertain 
first that NGOs did in fact reframe loss and damage as a concern of justice rather 
than as a primarily scientific or technical issue (the independent variable). Then the 
authors link this finding to evidence of framing and policy shifts in negotiations 
(the dependent variable) by tracing growing attention to loss and damage and the 
formation of coalitions on this issue (causal mechanisms or intervening variables) 
(Allan and Hadden 2017, 604). They use counterfactual analysis to argue against the 
competing explanation that the policy shift on loss and damage was attributable to 
a general increase in NGO advocacy rather than to the reframing of the issue (Allan 
and Hadden 2017, 614-615). 

Process tracing in this vein could be useful to help future research in deliberative 
democracy establish whether, for example, the inclusion of civil society, marginalized 
actors or experts enhances the deliberative quality of a decision-making process. This 
approach could also be used to explain the role of different actors in the diffusion of 
deliberative practices (as discussed in the previous subsection). 


Impacts of Deliberative Practices on Policymaking 
and Political Systems 


A third promising use of process tracing involves detecting the causal influence 
of deliberative practices or forums on policy decisions and broader political sys- 
tems. Papadopoulos (2012, 127) highlights this role for process tracing, noting that 
studying the impact of deliberative ‘devices’ or mechanisms ‘requires meticulous 
process-tracing, because participatory devices are only one of the inputs in the policy 
process, and their effects have a “diffuse and temporally dispersed character”.* The 
impacts of deliberative practices could be studied at various levels, including their 


ê Boswell and Corbett (2017) do not mention process tracing in their overview of how to compare 
deliberative systems, even though it could help to overcome two of the deficiencies they identify in existing 
research, namely rigid comparative frameworks and overly descriptive studies. 
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influence on a specific policy decision or their broader effects on the effectiveness of 
a political system (see also Vanhala 2017, 89). 

While assessing the impacts of mini-publics on deliberative systems has been a 
major theme of recent work in deliberative democracy (see, for example, Mackenzie 
and Warren 2012; Niemeyer 2014), empirical studies on this topic tend to use small-n 
comparative analyses without explicit resource to process tracing (see, for example, 
the two-case study in Felicetti et al. 2016, and the four-case study in Curato and Böker 
2016). Thus, it is necessary to look to other literature to find examples of how process 
tracing could illuminate this area. 

Process tracing has found productive use in investigating the effects of participa- 
tory or collaborative governance on environmental outcomes. Ulibarri (2015), for 
example, examines the role of collaborative governance in licensing hydroelectric 
power generation in the United States. Her analysis involves a comparative study 
of seven licensing processes, chosen to span cases of high and low collaboration. The 
comparative analysis reveals a positive correlation between levels of collaboration 
and the stringency of environmental standards contained in hydropower licences but 
is insufficient to demonstrate a causal relationship between the two. To address this, 
Ulibarri conducts process tracing on a subset of three cases to explore causal mecha- 
nisms through which collaboration may have influenced the licensing requirements. 
In the high-collaboration case, she finds that ‘stakeholders used a careful process of 
discovery and deliberation to work toward a comprehensive settlement agreement, 
and sought to include all interests each step of the way’ (Ulibarri 2015, 294). In 
the low-collaboration case, the government authority largely sided with the electric- 
ity utility applying for the licence despite calls from stakeholders—which had been 
marginalized in the licensing process—to include a wider range of licensing require- 
ments (Ulibarri 2015, 294). Although the process tracing results are presented quite 
briefly, they provide a plausible basis to support the conclusion that the different 
outcomes were attributable, at least in part, to the extent of collaboration.’ 

With this example in mind, we can return to the question of how process tracing 
could strengthen analysis of the impacts of mini-publics on deliberative systems. A 
first step would involve systematic categorization of the range of observable impacts 
that mini-publics could have. Goodin and Dryzek (2006, 219) present a useful typol- 
ogy of eight kinds of deliberative impact: ‘actually making policy, being taken up in 
the policy process, informing public debates, market-testing of proposals, legitima- 
tion of public policies, building confidence and constituencies for policies, popular 
oversight, and resisting co-option. Process tracing would require generating a set of 
empirically observable indicators associated with these impacts as well as a typol- 
ogy of causal mechanisms through which mini-publics could influence each of these 
indicators. Taking as an example the role of mini-publics in informing public debate, 


° A further example of how process tracing can be used to assess the effects of participatory governance 
mechanisms on democratic quality can be found in Friedrich (2006), who analyses the role of the European 
Unions Open Method of Coordination in social inclusion policy in Germany. 
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indicators could include public awareness of the mini-public’s findings or of the range 
of available policy options, or public opinion on or interest in the topic (Goodin 
and Dryzek 2006, 228). Causal mechanisms could include different types of com- 
munication channels (e.g. mainstream media reporting, social media, or word of 
mouth), the involvement of different individuals or organizations in those channels 
(e.g. mini-public participants or organizers, civil society organizations or policymak- 
ers), the level of financial resources invested in disseminating findings, or contextual 
factors (e.g. pre-existing levels of public awareness about the topic).*° Process tracing 
could then be used to assess the role of these causal mechanisms in producing wider 
deliberative impacts. 

This strategy could, in turn, help to resolve some ongoing puzzles in exist- 
ing empirical analysis of mini-publics. For example, Curato and Boker (2016) 
attribute the limited impact of the Australian Citizens’ Parliament in part to the fact 
that it coincided with catastrophic bushfires in Victoria, which may have diverted 
media coverage away from the Parliament. However, another case they analyse— 
the Conférence de Citoyens in France—attracted widespread public attention despite 
coinciding with the soccer World Cup hosted in France at that time. This suggests 
that one causal mechanism that may work against the impact of mini-publics (i.e. the 
presence of competing priorities for public attention) may only have limited leverage 
in explaining their impact. Process tracing could investigate whether other factors 
may have been more important in explaining the level of public attention in each 
case (e.g. the choice of topic or the level of investment of each process in media 
engagement). 

Finally, this mode of process tracing could help to investigate other contro- 
versial questions in deliberative democratic theory, including whether and how 
non-deliberative acts, such as protest, can function as a trigger for enhanced delib- 
eration. One challenge with using process tracing to detect systemic impacts is that 
the influence of a single mini-public may be difficult to detect amidst the cacophony 
of deliberative and non-deliberative communication in the system. This suggests that 
process tracing of broader impacts may require redefining the scope of the case study. 
Rather than the case comprising a one-off mini-public, it may be necessary to treat 
a broader programme of deliberative innovation as the case (e.g. a legislated pro- 
gramme of deliberative processes such as the one discussed in Ravazzi (2017) above) 
and to conduct an assessment of the cumulative impacts of the programme. 


Conclusion 


In this chapter, I have argued that process tracing has a valuable role in the repertoire 
of methods for assessing deliberation. Process tracing cannot by itself confirm or 


1° In a quantitative study of the impacts of deliberative processes on political efficacy, Geissel and Hess 
(2017) identify several factors (e.g. decision makers’ support for the process and dedicated staff resources 
to run deliberative processes) that could also be used in process tracing. 
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undermine some of the foundational normative claims of deliberative democracy— 
such as the intrinsic value of citizen deliberation—but it can shed light on contested 
empirical claims about the effects of deliberation and thus perform a valuable role 
in constructing and testing falsifiable ‘middle-level’ theories of deliberation (Mutz 
2008). The examples presented here have shown how process tracing can illumi- 
nate key questions facing deliberative theorists, including how deliberative practices 
are adopted, how participants influence the outcomes of deliberative processes, and 
how those outcomes in turn influence broader political dynamics. Future research on 
these questions could begin by mapping out the explanations offered by competing 
theories of deliberative democracy (drawing on other research in political science on 
sources of policy diffusion and institutional change) and systematizing the range of 
possible causal mechanisms that would support these explanations. Applying these 
frameworks effectively will require judicious selection of cases that offer opportuni- 
ties to test competing explanations, and creative combinations of single and small 
comparative process tracing studies with medium- and large-N research designs to 
maximize the explanatory leverage that process tracing can offer. 
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The systemic turn in deliberative democracy has drawn attention to the importance 
of deliberation outside of the confines of small group or facilitated deliberation (see 
Chambers 2009; Dryzek 2009; Hendriks 2006; Mansbridge 1999; Parkinson 2003; 
Parkinson and Mansbridge 2012). Whilst the conditions and limits of the deliberative 
system are still under discussion, attention has also moved towards how to study 
deliberative systems empirically (Ercan et al. 2017). Since methods used to assess 
micro deliberation are not always suitable for studying deliberation in the real world 
(Ercan et al. 2017, 197; Bachtiger and Parkinson 2019, 3), researchers must expand 
their methodological repertoire to meet this challenge. 

Q methodology is an approach to studying intersubjectivity (Brown 1980) that can 
provide insights into both micro deliberation and deliberative systems. A Q study 
asks subjects to rank a set of statements or images on a matrix, and uses qualitative 
and quantitative techniques to identify correlations and establish shared viewpoints. 
Here I discuss how Q method can be used to study micro deliberation, followed by 
how it can enhance our empirical understanding of the deliberative system through 
identifying discourses in the public sphere and giving an insight into deliberative 
quality on a discursive level. I also provide an overview of the assumptions under- 
pinning the method, the practicalities of conducting a Q study, and consider what Q 
can and cannot do. 


Theoretical Assumptions 


Ercan et al. (2017) argue that interpretive research methods have an important role 
to play in the study of deliberative systems. In particular, interpretive methods seek 
to understand meaning and phenomena as they are socially constituted by the actors 
involved (2017, 202). Q is one such interpretive method. Ina Q study, participants are 
invited to rank a set of statements or images on a matrix, creating a map of their own 
viewpoint on the topic. The interpretation of these items is up to the participants; 
they make their own meaning (Brown 1997) and individuals’ subjectivity is taken 
seriously by Q methodology (Dryzek 2005b, 40). Items are ranked in relation to each 
other, and the overall map that makes up each ‘Q sort’ is a holistic account of that 
person's view. 


Lucy J. Parry, Q Methodology. \n: Research Methods in Deliberative Democracy. Edited by Selen A. Ercan, Hans Asenbaum, 
Nicole Curato, and Ricardo F. Mendonga, Oxford University Press. © Oxford University Press (2022). 
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The aim is to identify shared viewpoints in a way that takes into account the active 
construction of meaning by participants. The factors that emerge from a Q study 
are not predetermined by an externally imposed scale or definition (Cross 2005, 
211). Whilst the factor analysis employed to define clusters of individuals’ viewpoints 
reveals statistical relationships across individuals, Q methodology is grounded in 
interpretive principles (Dryzek 1990, 173). Analysis requires a close reading of qual- 
itative material gathered during the study, whilst ‘the statistical actions performed in 
Q methodology support us in interpreting the discrete subjective views on any given 
issue’ (Stevenson 2015, 6-7). 

John Dryzek (1990, 2005b) has also argued that Q is in keeping with the principles 
of deliberative democracy, given that it takes seriously the subjectivity of partici- 
pants (2005b, 40); the items sorted are presented in an accessible and transparent 
manner, and participants are free to construct their own meaning for each item, and 
the whole matrix. There is no assumption that participants will interpret each state- 
ment in the same way, since correlations are not made across statements. Indeed, 
the differential interpretation of the same statement may provide some of the most 
interesting findings. For example, in my study on foxhunting, I found divergent 
interpretations of the statement ‘I consider myself to be an animal lover’. One pro- 
hunting viewpoint ranked this statement with the highest level of agreement, with 
participants pointing out that foxhunting supporters are animal lovers, despite per- 
ceptions. Another pro-hunting discourse was more cautious, suggesting the term 
‘animal lover’ implied somebody who wanted to cuddle up with animals, which is not 
in their (perceived) best interests. On the anti-hunting side, the most critical animal 
rights position also had a lukewarm perception, with stronger sentiment attached 
to more abstract statements on the morality and politics of hunting (Parry 2016, 70; 
Parry 2019). 


Q Methodology and Micro Deliberation 


During deliberative forums, surveys are sometimes taken to assess participants’ views 
before and after deliberation. This enables us to understand if and how an inter- 
vention affects peoples’ views—although whether it is deliberation itself that affects 
preference change is contested (Goodin and Niemeyer 2003). Q methodology can be 
utilized for a similar purpose, to assess the extent to which participants’ preferences 
shift following deliberation. However, Q offers an alternative approach to a stan- 
dard survey, in identifying synthesized viewpoints rather than aggregating individual 
viewpoints (Pelletier et al. 1999, 105). 

Indeed, a good start to understanding Q methodology is by taking surveys as a 
point of comparison. Q uses an inverted form of factor analysis that correlates indi- 
viduals as variables, rather than survey questions as variables (Van Exel and De Graaf 
2005, 1). Whilst in a survey, questions and a scale are constructed by the researcher, 
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participants in a Q study arrange a set of items (a set of statements or images termed 
the ‘Q set’) on a matrix in an order that is significant to them (Brown 1980, 6). How 
to interpret and rank each item is down to the participant rather than the researcher. 
Factor analysis then identifies relationships across participants’ subjectivities, pro- 
ducing factors that ‘refer to groups of people that sorted the statements in a similar 
way (Pelletier et al. 1999, 108; emphasis in original). 

In the context of pre/post deliberation, Q can identify opinion shifts through 
assessing the extent to which an individual is associated with a discourse before and 
after the event (e.g. Niemeyer 2011; Pelletier et al. 1999). There are a variety of tech- 
niques that can be used to understand these changes, depending on the purpose of 
the research. Simon Niemeyer (2011, 2019) deploys surveys alongside Q method- 
ology to understand ‘intersubjective consistency —the extent to which individuals’ 
underlying subjective will (according to Q data) aligns with their policy preferences 
(according to survey data). Niemeyer argues that the discourses resulting from Q 
analysis constitute participants’ ‘underlying reasoning regarding the issue at hand’ 
whilst survey data provides participants’ ‘expressed preferences’ (2011, 104). Follow- 
ing deliberation at two citizens’ juries, he finds intersubjective consistency is higher; 
that is to say, the deliberative process resulted in a closer alignment of both types of 
preference, Q and survey preferences. 

Pelletier et al. (1999) deploy a range of analytical strategies to assess the effects 
of deliberation on the food system using Q method. They analyse differences in the 
substantive content of the shared viewpoints before and after a series of participa- 
tory planning processes, as well as how individual membership of each discourse 
is altered following the events. Their findings indicate that the factors (viewpoints) 
produced are remarkably stable—containing the same substantive themes before and 
after deliberation (Pelletier et al. 1999, 113). They also analyse the extent to which 
individuals associated with one factor become associated with another following 
deliberation. A more subtle change can also occur if individuals remain associ- 
ated with the same factor, but the strength of that association changes. This type of 
fine-grained analysis allows an insight into subtle shifts in viewpoint that might not 
be expressed explicitly through preference ranking in a survey. 


Q Methodology and Deliberative Systems 


The deliberative systems approach—though diverse and dynamic in its 
conceptions—holds a common emphasis on the importance of deliberation across 
and within different sites of interaction (e.g. Dryzek 2009; Hendriks 2006; Mans- 
bridge 1999; Mansbridge et al. 2012). One response to the challenge of studying delib- 
erative systems empirically is an emphasis on discourses and/or narratives (Boswell 
2016; Ercan et al. 2017, 201; Stevenson and Dryzek 2014). Dryzek’s (2005a, 9) 
definition of a discourse is congruous with the ‘structure and form’ (Brown 1986, 58) 
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of the shared viewpoints produced by Q methodology through the process of factor 
analysis and interpretation: 


... a Shared way of apprehending the world. Embedded in language, it enables 
those who subscribe to it to interpret bits of information and put them together 
into coherent stories or accounts. Discourses construct meanings and relation- 
ships, helping to define common sense and legitimate knowledge. 

(Dryzek 2005a, 9) 


How a participant ranks the items in a Q study reflects ‘the way a particular indi- 
vidual, in particular circumstances and at a particular time, relates to, and forms 
conceptions of, certain aspects of the world’ (Barry and Proops 1999, 338); a Q 
methodological analysis clusters together those individuals to reflect intersubjective 
viewpoints or discourses. 

Proponents of the discursive approach to deliberative systems suggest that it is 
the constellation and contestation of discourses that denotes a healthy delibera- 
tive public sphere (Curato et al. 2013; Stevenson and Dryzek 2014). Other sites in 
the system—such as ‘empowered space’ (Dryzek 2009) where authoritative deci- 
sions are made—are evaluated according to the extent to which public discourses 
are reflected (e.g. Boswell 2016; Parry 2016; Stevenson and Dryzek 2014). The trans- 
mission mechanisms that connect discrete sites can also be examined to determine 
which discourses travel through the system and how they are communicated, as well 
as considering why some discourses are successfully reflected in policy decisions, 
whilst some viewpoints are marginalised, or expressed in some sites but not others 
(e.g. Boswell 2016; Boswell et al. 2016; Parry 2016; Stevenson and Dryzek 2014). 

Whilst the inclusion of a diverse range of viewpoints is not sufficient to warrant a 
healthy deliberative system (Ercan et al. 2019; Owen and Smith 2015), studying and 
mapping discursive elements within and across different sites of interaction makes a 
valuable contribution to the understanding of deliberative democracy at both macro 
and systemic levels (Ercan et al. 2017). It allows us to address two key questions: iden- 
tifying ‘the sites, agents and discursive elements that comprise a deliberative system’ 
and the ‘connections and transmissions across different sites of a deliberative system 
(Ercan et al. 2017, 197). Q can identify discourses that can be traced through different 
sites in the deliberative system to help us understand the deliberative or democratic 
deficits present through the suppression or distortion of certain discourses. 

Q may also contribute to the evaluation of the deliberative system. David Owen 
and Graham Smith (2015) argue against relying on the balance of discourses as a 
marker of deliberative quality, pointing out that evaluation on a systemic level could 
result in an apparently healthy system that is made up of non-deliberative actions. 
They point out that assessing the balance of discourses does not tell us anything 
about the deliberative quality of actual interactions within a given site in the system. 
Nonetheless, Q may still be able to provide some insight into deliberative quality. The 
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viewpoints resulting from a Q study can reveal potential points of consensus or dis- 
tinctions that may not be immediately apparent in other accounts of a debate (Brown 
1996, 17). This is particularly relevant for highly polarised debates, where Q can 
contribute complexity to the usual dichotomies. My study on foxhunting found two 
viewpoints on animal rights and animal welfare that were highly correlated. This has 
deliberative significance because it shows that two perspectives, often characterized 
as mutually exclusive, actually have more in common than not, and that individuals 
may subscribe to both views in different measures (Parry 2019, 20). 

By looking at the discursive content Q produces, it is also possible to examine 
whether discursive commitments and characteristics enable or preclude possibilities 
for inclusive deliberation (Parry 2016, 78). If a discourse is intrinsically antagonistic 
or dismissive towards alternative viewpoints, possibilities for meaningful reflection 
and listening across discourses may be hindered. Discourses can constrain and enable 
actors, realizing a kind of interdependent agency along with actors: discourses shape 
actors as actors shape discourses (Leipold and Winkel 2013). Whilst this cannot 
make any definite conclusions about the health of the public sphere—since Q can- 
not determine who is saying what, how, and where—it can provide some insight into 
deliberative quality at the discursive level. 


Doing Q Methodology 


It is worth mentioning that to many proponents, Q methodology is much more than 
just doing a Q study. It encompasses its founder William Stephenson’s determina- 
tion to develop a scientific approach for studying subjectivity (Watts and Stenner 
2012, 25). Stephenson, who held doctorates in physics and psychology, drew upon 
and contributed a variety of theories throughout his career that elaborated on his ini- 
tial inverted factor analysis technique (Stephenson 1935), including behaviourism, 
concourse theory, and constructivism.’ However, it is not strictly necessary to 
acquire in-depth knowledge of this background in order to conduct a successful Q 
study (Watts and Stenner 2012, 5). 

The first practical stage is selecting a suitable research question to define the scope 
of the study. To identify discourses within the deliberative system or public sphere, 
this may be as straightforward as asking ‘what are the discourses that exist on X? Q 
studies can be conducted to address more complex questions, such as asking partic- 
ipants to respond to a policy issue or to share their personal experience of particular 
circumstances (Watts and Stenner 2012, 55-56). The question must be clear and 
unambiguous, given that it will not only guide participants but will also establish 
the scope of the study. 


* Watts and Stenner (2012) provide a digestible chapter on Stephenson's theoretical trajectory over his 
career. 
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The statements sorted by participants (the ‘Q set’) are designed to be representative 
of the landscape of debate (known as the ‘concourse’). The concourse can be devel- 
oped in a variety of ways depending on time, resources, and research aims. The ‘Q set’ 
is usually comprised of written statements, but it can also be visual, eliminating any 
literacy barriers to participation (e.g. Alexander et al. 2018; Hardy and Pearson 2016). 
It is possible to use interviews as the basis for the concourse; for my research on fox- 
hunting, I conducted ten interviews from which I extracted 248 potential statements 
structured according to the central themes of the foxhunting debate (identified from 
existing literature): people, animals, place, and politics (Parry 2019). Statements can 
also be generated from existing material including literature, blogs, reports, or media. 
Using published material saves time, and interviewing may not give additional value 
if a wealth of published material already exists (Stevenson 2015, 3). However, inter- 
views offer the advantage of providing statements in readily accessible language and 
providing insights that might not be available in existing material; if you are inter- 
ested in marginalized viewpoints, interviews might provide insight not covered in 
existing material. 

From here, you need to develop the ‘Q set’ from the concourse. Again, methods 
vary—from highly structured, theory-driven sampling approaches (see Dryzek and 
Berejikian 1993, 51) to more fluid ones where the only requirement is that the ‘Q 
set’ is comprehensive and broadly representative of the concourse (Brown 1986, 73). 
Either way, this is an iterative process aiming to arrive at a manageable number of 
items. There is no hard rule for the exact number, but more statements will require 
more time for sorting and may be overwhelming for some participants (Watts and 
Stenner 2012, 61). However, there must be a sufficient range and number to represent 
the concourse. The representativeness of the ‘Q set’ can be checked through piloting 
the study, checking for noticeable gaps, and ensuring that participants feel fully able 
to express their view. 

Alongside the refinement of the ‘Q set’, you will need to decide a matrix for par- 
ticipants to rank the statements on. For the most part, this comprises a quasi-normal 
distribution as illustrated in Table 21.1. 

Items placed towards the centre of the scale are generally less crucial to the partic- 
ipant’s viewpoint—although this is not always the case,” whilst items at the extremes 
are more important, either positively or negatively. 

The next stage is selecting participants. Whilst survey participants are often 
selected to be representative of a broader population, in Q it is the ‘Q set’ that ful- 
fils this role. The selection of participants in Q is like the selection of questions in a 
survey—it is done carefully and strategically. The aim is to have a group of partic- 
ipants who are likely to have interesting, relevant, and diverse views on the subject 
(Watts and Stenner 2012, 70). It is not necessary to have a particularly large number 


There may be statements placed towards the centre that participants feel entirely neutral about, feel 
torn about, or cannot decide how they feel about, or they simply do not feel knowledgeable enough to 
stake a stronger opinion on. These feelings can be discussed during the Q sort and the researcher can take 
note as it may aid in interpretation later on. 


Lucy J. Parry 313 


Table 21.1 Quasi-Normal Distribution 


Most Disagree Most Agree 


PECCAR 


Source: Parry (2016, 68). 


of participants, since the aim is not to make claims about generalisability to the 
wider population, but to have sufficient variety to establish the existence of shared 
viewpoints. 

In my foxhunting study, I selected campaigners, activists, animal protection pro- 
fessionals, hunt staff, and people who took part in hunting or who had some direct 
experience with hunting. In studies such as Niemeyer (2011) or Pelletier et al. (1999), 
studying the views of participants in a deliberative process, this kind of purposive 
sampling will likely not apply because participants are the people taking part in 
the deliberative process. The precise composition of the participants will therefore 
depend on the purposes of the study and research question. Q methodology cannot— 
and does not—claim generalisability in the same way that a survey deployed to a 
representative sample of the population might. Q cannot claim what percentage 
of the population subscribe to a particular viewpoint, or what a particular demo- 
graphic thinks. A Q study may not even collect demographic information about 
study participants, given that its focus is on what is being said, rather than who 
is saying it (Parry 2016, 15). However, it is possible to claim a different type of 
generalizability from Q, that of ‘substantive inference (Thomas and Baas 1992, 22) 
about a topic. Rather than generalizing about who holds certain viewpoints, it is 
possible to generalize about the nature and substance of the viewpoints themselves. 
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Moreover, as Dryzek (2005b, 205) points out, ‘each factor is a generalization because 
it indicates how people of a particular kind think. 

The administration of the study—the ‘Q sort’-—can be carried out online or in 
person. Online has the advantage of convenience and there are a number of soft- 
ware packages for conducting Q studies, some of which also carry out the analysis. 
However, given that Q is usually novel for participants, doing it in person gives you 
the opportunity to provide guidance and answer questions. I find that some partic- 
ipants like to discuss each statement, often providing additional material that aids 
interpretation. It is possible—and desirable—to conduct interviews after the sort- 
ing to ask about participants’ reasoning. Being present also helps when participants 
become frustrated or confused about the process—I have occasionally had people 
confuse the poles of the matrix, or feel restricted by the distribution. Talking this 
through in person helps ensure that participants feel able to complete the sorting 
procedure thoughtfully, rather than giving up or going through it very quickly. Of 
course, this takes time for both parties; I have spent up to three hours with a single 
participant following an unstructured, conversational approach. Whilst this gleaned 
valuable qualitative material that assisted in the interpretation of factors later on, it 
is not always feasible. 

Factor analysis can be carried out in a range of software packages, although PQ 
Method (Schmolck 2014)—a Q-specific, command-line programme developed in 
the 1990s—remains a popular option. It is free, relatively simple to use, and carries 
out only factor extraction and rotation for Q methodology. PQ Method offers two 
types of factor extraction—Centroid and Principal Components Analysis. Their rel- 
ative merits are a matter of debate (see Akhtar-Danesh 2017; Ramlo 2016), but in 
reality they produce similar results (Watts and Stenner 2012, 99). The first stage is 
factor extraction, which entails extracting ‘portions of shared or common meaning’ 
(Watts and Stenner 2012, 98) from the matrix of all the relationships in the study. Fol- 
lowing extraction, you must decide how many factors to retain. This can be informed 
by a range of statistical criteria (see Watts and Stenner 2012, 105-110) but should also 
be aided by your knowledge and judgement (Brown 1980). 

Following extraction, factors can be rotated. Brown (1980, 224-226) provides a 
useful analogy for this process. Imagine a transparent sphere containing various dots 
embedded inside. The constellation of dots look different depending on how you turn 
the sphere in your hand. Rods can be inserted at right angles into the sphere; these are 
the factor axes. Rotation is akin to examining the dots from different angles; the dots 
represent individual ‘Q sorts. Their position does not move, but you can look at them 
from different perspectives. The closer a factor axis is to each Q sort the more closely 
a participant is associated with that factor. Factor rotation ‘aims to position each fac- 
tor so that its viewpoint closely approximates the viewpoint of a particular group 
of Q sorts, or perhaps just one or two of particular importance’ (Watts and Stenner 
2012, 127). Following rotation, my research on foxhunting revealed four groups of Q 
sorts: two broadly in favour of and two against foxhunting. My analysis shows that 
whilst the two anti-hunting positions are highly correlated statistically, they are dis- 
tinct (Parry 2019, 8). Distilling this kind of nuance required the use of qualitative 
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material and judgement; this highlights the importance of an interpretive approach 
in Q analysis, rather than relying solely on the quantitative data. 

The final stage in the analysis is the creation of ‘factor arrays. Factor arrays are 
a hypothetical Q sort—what it would look like if a person had a 100 per cent load- 
ing on that factor. Factor arrays are compiled from individual ‘Q sorts’ that are highly 
associated with a given factor. Factor arrays always differ slightly from actual ‘Q sorts, 
since a factor array ‘is simply a best-possible estimate of the factor’s viewpoint’ (Watts 
and Stenner 2012, 143; emphasis in original). These form a tangible qualitative foun- 
dation for interpretation. The final output from PQ Method includes a potentially 
overwhelming amount of additional data, including Z scores, factor correlations, and 
arrays of statements across factors from consensus to disagreement. Any to all of 
these components can be used for interpretation and further analysis, depending on 
your research aims and level of comfort with statistics. Interpretation can be further 
supported by interviews or additional information from participants. This helps to 
flesh out factor arrays into substantive viewpoints with quotes and examples, bringing 
discourses to life. 

Many Q studies simply identify and discuss a range of shared viewpoints at length. 
For those aiming to analyse the discourses before and after a deliberative interven- 
tion, this will only be the first stage. Either way, it is helpful to provide a narrative 
description of each viewpoint including both the wording of the statements and how 
they are scored in each factor array, and including interview material for further 
elaboration: 


Hunting is nothing more than inflicting gratuitous cruelty on animals for the sake 
of entertainment (S22, +3) and there is no place for hunting in a modern, civilized 
society (S50, +5) because ‘as with other modern attitudes, particularly 21st cen- 
tury, we’ve moved into a different place of awareness as regards animal life ... 
there is no place for taking pleasure in hunting another species, particularly where 
there’s no necessity involved in it . . . apart from gaining pleasure’ (P23). 

(Parry 2019, 11) 


Niemeyer (2011) provides diagrams that indicate how each factor relates to others, 
along with a visual plot of how the distribution of ‘Q sorts’ changes following deliber- 
ation. I have personally presented the results of a rough and ready Q study as spoken 
word poetry at a Hackathon, where time was short and I wanted to engage a non- 
academic audience (Carson and Parry 2016). The precise presentation will depend 
on the purposes of the research, but it is worth noting that as Q is unfamiliar to many, 
it may require some explanation in a publication. Here Watts and Stenner (2012) are 
particularly helpful in providing practical guidance for writing up and publishing Q 
studies. 
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Conclusion 


Q methodology can enhance our understanding of deliberative democracy in a num- 
ber of ways. It offers the opportunity to assess deliberation in the context of both 
micro deliberation and deliberative systems. At the macro level, Q can be used to 
identify shared viewpoints in the public sphere, which in turn provide the discur- 
sive foundations that can be traced within and across the system. Q can also provide 
insight into deliberative quality through assessing the extent to which the resulting 
discourses contest and overlap. 

Q methodology also reaches across typical methodological divides through its use 
of both qualitative and quantitative techniques, although it remains an interpretive 
approach commensurate with the principles of deliberative democracy: it takes seri- 
ously the subjectivity and agency of individuals in expressing their views. This makes 
it an important empirical arsenal for scholars interested in studying deliberation and 
deliberative democracy, for both understanding the constituents of the deliberative 
system and assessing deliberative quality on a discursive level. 
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Dramaturgical Analysis 


Selen A. Ercan and Carolyn M. Hendriks 


At first glance, drama and deliberation would appear unlikely bedfellows. Con- 
ventionally, deliberation is conceptualized as a communicative process centred on 
reasoning and authenticity, not spectacle and drama. Yet the world of theatre and its 
associated concepts of scripting, setting, staging, and performance can offer a rich 
palette for analysing public deliberation. In this chapter, we introduce dramaturgi- 
cal analysis as a useful analytical framework for studying the performative aspects of 
public deliberation, drawing on examples to illustrate its application. 

Dramaturgy is a sociological perspective about how people construct, present, and 
manage themselves in social interactions. It is premised on the notion that when peo- 
ple interact, they are part of a performance that shapes the way they talk, engage, and 
act. In theatre, dramaturgy is used to examine how moving bodies, voice, sound, 
and light can tell a story and affect values (Szatkowski 2019). It was Erving Goffman 
(1959), who first adapted the term dramaturgy from theatre to sociology to study how 
people conduct themselves in social interactions. Since then, dramaturgy has been 
applied in diverse fields of social sciences (for overviews, see Travers 2001 and Birch 
2017). In sociology, it is usually associated with ‘symbolic interactionism’ which is 
a theoretical perspective exploring how society is created and maintained through 
repeated interactions between individuals (Carter and Fuller 2016). In political sci- 
ence, dramaturgy has been used to study the performances of political leaders and 
politicians (Merelman 1969), the role of symbols (Edelman 1964), the staging of 
propaganda rallies (Mayo Jr 1978), and social movements (Benford and Hunt 1992). 

Dramaturgical analysis is an interpretive analysis which draws on the vocabulary 
of dramaturgy (such as theatre, performance, audiences, staging, scripting, acting) 
to describe and make sense of what is going on in social, cultural, and political life. 
Dramaturgy enables researchers to see politics as ‘a sequence of staged performances’ 
and to focus empirically not only ‘what people say’ but ‘how they say it, ‘where they 
say it, and ‘to whom they say it (Hajer 2009, 65). By undertaking dramaturgical anal- 
ysis, researchers can develop a better understanding of how political controversies are 
enacted, how different acts relate to each other, and how conflicts are expressed and 
resolved. 

There are many ways dramaturgical analysis can be applied to the empirical 
study of public deliberation, understood as a broad communication process taking 
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place in structured forums and in the broader public sphere (Ercan and Dryzek 
2015). Dramaturgical analysis directs the researcher's attention to often-overlooked 
or taken-for-granted aspects of public deliberation, such as the performative styles 
and body language of the actors involved in deliberation, where they stand, how they 
enact and stage their arguments, what symbols and artefacts they use to reinforce 
their viewpoints, and how they reach out and persuade diverse audiences. By asking 
these questions, researchers can develop novel interpretations of what is going on 
in and around deliberative processes and establish new connections between previ- 
ously unexplored aspects of public deliberation, such as the relationship between the 
physical setting and the quality of communicative interactions. 

In this chapter, we first offer an analytical framework that can be used for inter- 
preting public deliberation through the lens of dramaturgy. We then present three 
illustrations of dramaturgical analysis in deliberative democracy research. Our aim 
is to encourage scholars of deliberative democracy to be more attentive to the perfor- 
mative aspects of deliberation and to build research knowledge on the components 
and practice of dramaturgical analysis. 


A Framework for Dramaturgical Analysis 


For researchers intending to use dramaturgy as part of their empirical work, the first 
step involves developing or adapting a framework for the dramaturgical analysis. 
There are various dramaturgical frameworks that scholars have developed to cap- 
ture the performative aspects of political practices (for examples, see Benford and 
Hunt 1992; Hajer 2005a; Alexander 2011; Hendriks et al. 2016; Yuana et al. 2020). 
These frameworks serve to ‘operationalize’ various theatrical concepts and provide 
researchers with scaffolding for organizing and interpreting their data. 

In this chapter, we elaborate on one particular dramaturgical framework that was 
originally developed by Maarten Hajer (2005a) to study the performative aspects of 
planning and policymaking more broadly. Hajer applies dramaturgy to the ‘perfor- 
mativity’ of language (how it does something) as well as the ‘performance’ of language 
(how it can convey certain meanings that are constantly reproduced and enacted in 
a particular setting). His analytic framework includes four partly overlapping dra- 
maturgical dimensions for generating and analysing data: scripting, setting, staging, 
and performance. 

Scripting refers to the intentional design of the entire interaction. It is about devel- 
oping the cast of characters and the audience, how they ought to perform or interact, 
and who should be included and excluded. It also defines the characteristics of the 
central participants as either active or passive, interested or disinterested, collabora- 
tive or antagonistic, competent or incompetent. Apart from the characters and cues 
on the frontstage, scripting captures a host of frontstage and backstage activities. In 
his dramaturgical analysis of the work undertaken by citizen engagement practition- 
ers, Oliver Escobar (2015, 278) observes how scripting ‘assembles time (e.g. pacing, 
opportunity), space and dynamics (e.g. layouts, formats), characters (e.g. individuals, 
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groups, places), strategies and tactics (e.g. exposing participants to diverse others), 
materials and artefacts (e.g. tablecloth, facilitation tools), narratives and frames ... 
and enactments (e.g. facilitating, orchestrating)’. Scripting in this broader sense func- 
tions as a kind of meta-category, under which Hajer’s (2005a) three other conceptual 
categories sit, namely setting, staging, and performance. Pertinent to the category of 
scripting is also the idea of a ‘counter script’ which refers to ‘the conscious activity of 
antagonists who try to alter the effects of particular stagings of politics, to twist the 
meaning of what is said by giving it a new contextualization or introducing a new 
antagonism. (Hajer 2009, 66). 

Setting refers to the physical environment or context in which interaction takes 
place, and how it shapes behaviour and communication. Settings not only shape 
what is considered appropriate and expected behaviour in a given context, but 
they also configure expectations about dress code and communication. A setting 
can include the scenes, or backdrops, against which a communicative interaction 
occurs, such as a parliament, a shopping mall, or a particular landscape. A set- 
ting can also encompass the props and artefacts that shape the interaction. These 
include things such as seating arrangements, location of speakers, use of images, 
tabled reports, and minutes of previous meetings. All settings consist of a backstage, 
where individuals prepare for a performance, and a frontstage where they are on 
display. 

Staging is about managing and directing the performance, particularly the interac- 
tion between participants. Staging typically involves drawing on existing symbols or 
inventing new ones. An important aspect of staging is the distinction between those 
central to the performance (such as the directors and the actors) and the audiences 
(Hajer 2005a). Staging draws analytic attention to questions about who is coordinat- 
ing the entire performance and how, and who is the intended or imagined audience. 
From a dramaturgical perspective, audiences are not simply out there waiting to 
be engaged and entertained by the actors. Instead, they are actively constructed as 
part of the performance (Turnhout et al. 2010; Hajer and Uitermark 2008). In this 
sense, audiences resemble publics, which do not exist as an entity, but must always 
be ‘brought into being’ through various performative practices and actions (Butler 
2010, 147). 

Finally, performance is concerned with how staged actions and interactions con- 
struct new understandings of the issues at stake and reconfigure power relations. 
It is about the way in which social realities are produced, reproduced, or chal- 
lenged (Polletta 2006). Performance, however, is an interpretive act and it is not 
always clear how an interaction or activity will be received, and what messages 
and power relations it will reshape or affect. Sometimes performances can gener- 
ate unexpected or unintended effects, especially if they unfold rapidly (Hajer and 
Uitermark 2008). 

Table 22.1 below outlines an illustrative list of empirical questions associated with 
the four different dimensions of Hajer’s (2005a) dramaturgical framework. These can 
be adapted and applied to study different settings and aspects of public deliberation. 
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Table 22.1 Four Dimensions of Dramaturgical Analysis 
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Dramaturgical Definition Empirical questions for deliberative 

dimension democracy research (illustrative list 

only) 

Scripting The overall design of the inter- e How did interactive space come 
action which determines what into being, and what is its goal, 
characters are in the performance and central message? 
and how they ought to act and for e How do the organizers describe 
what end. and promote the interaction, 

and its purpose? 

e Who are the intended char- 
acters and what acting cues 
(instructions) are they given? 

Setting The physical situation or context e What is the ‘physical’ nature of 
in which the interaction takes the interactive space 
place, including the building, e What artefacts, props, images 
backdrop and backstage, and any and symbols are used? 
artefacts and props that are used. e How does the particular setting 

influence how actors behave, 
what is said, what can be said, 
and what can be said with 
influence? 

Staging The management and direction e How are interactions within the 
of the interaction, as well as dis- interactive space organized, and 
tinguishing between different facilitated? 
participants and audiences. e Who are the participants? What 

roles do they play? 

e Who are the audiences? How 
are they constructed? How do 
they interact (if at all) with the 
performance? 

Performance The way in which the contextu- e What function does the inter- 


alized interaction itself produces 
social realities such as under- 

standing of the problem at hand, 
knowledge, new power relations. 


active space play in the broader 
political debate? 

To what extent does the inter- 
active space shape how issues 
are defined and discussed, and 
how knowledge is used and 
understood? 

How does the performance 
reinforce or reconfigure power 
relations? 


Source: Adapted from Hajer (2005a). 


Once the analytical framework and associated empirical questions are developed, 
the next step in a dramaturgical analysis involves gathering and organizing the ‘data. 
Data for dramaturgical analysis is typically obtained through the close observation 
(either in real-time or via video recording) of social or political interactions and 
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the behaviour of actors in both frontstage and backstage (e.g. Escobar 2015). For 
example, in studying a deliberative forum, one might observe the physical and aes- 
thetic aspects of the process (the form of the invitations, the seating arrangements and 
the use of space, colour, symbols, and music), how the actors engage on the stage (for 
example, their body language, clothing, tone of voice, and how and who they interact 
with), and what goes on behind and outside the forum (the backstage happenings, 
the real-time media, and political developments). 

Observational data can be supplemented with interviews, media articles, Internet 
sources and other relevant documentation that helps the researcher to reconstruct 
the ‘deliberation in action’ as a series of staged performances. Data gathering in 
dramaturgical analysis must support ‘the discovery of communication, symbols, ges- 
tures, facial expression, props, sentiment, documentation and other instruments of 
human interaction (Corrigan and Beaubien 2013, 309f.). Interviews with the actors 
involved in the performance or its audience can be especially fruitful to help flesh out 
the meaning of certain acts and symbols and identify how actors themselves experi- 
ence and relate to the performances at work. Interviews with various actors beyond 
forum participants, including organizers and facilitators, and even non-participants, 
can enable the researcher to capture diverse interpretations of the performances 
under scrutiny. 

Dramaturgical analysis can also be combined with other methods of analysis such 
as discourse analysis (for examples, see Hajer and Uitermark 2008; Hendriks 2009; 
Yuana et al. 2020). In this case, each analysis draws on different kinds of data: for 
example, a discourse analysis might draw primarily on written and spoken text (such 
as policy documents, media articles, interviews, and recordings), whereas a comple- 
mentary dramaturgical analysis might take into account non-verbal aspects of the 
interaction, shifting the researcher’s focus on the performative effects of the setting 
and the use of imagery, colour, or props. The discourse analysis serves to ‘identify 
how certain terms (concepts, classifications) dominate a political debate over a period 
of time, and the dramaturgical analysis implies a more precise focus on how people 
use particular terms in particular situations’ (Hajer 2009, 54). It is also possible to 
combine dramaturgy with concepts from geography and environmental justice (e.g. 
Barnett and Scott 2007), science and technology studies (e.g. Felt and Fochler 2010), 
narrative analysis (e.g. Roberts 2018), and interpretive policy analysis (e.g. Escobar 
2015). 


Dramaturgy of Public Deliberation: Illustrative Examples 


Dramaturgical analysis enables researchers to investigate often-overlooked 
or taken-for-granted aspects of deliberative practices. These include the 
physical environment within which deliberation takes place and how it 
influences behaviour and communication; how the seating arrangements in 
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deliberative forums affect the way participants relate to each other; and how the 
materials and artefacts used in deliberative practice shape the overall atmosphere as 
well as the prospects for interaction. Furthermore, dramaturgical analysis offers a 
powerful way of studying the non-verbal expression in public deliberation, which 
has been a key topic of scholarly interest and investigation in the field (e.g. Rollo 
2017; Curato 2019; Mendonga et al. 2022; Hendriks et al. 2020). 

Below, we draw on some specific examples to illustrate the ways dramaturgical 
analysis can be used in the empirical study of deliberative democracy. Given our 
understanding of public deliberation as a broad communicative process that goes 
beyond structured forums, our examples come from a variety of spaces where delib- 
eration takes place in contemporary societies, including the deliberative forums, 
the public sphere encompassing both physical and digital spaces and deliberative 
policymaking processes. 


Dramaturgical Analysis of Deliberative Forums 


Dramaturgical analysis can be applied to empirically study deliberative forums, 
which have been the key focus of attention in the field of deliberative democracy 
especially over the past two decades. Scholars study these forums so that they can 
identify and optimize the conditions that encourage high-quality deliberation. They 
develop innovative procedures to select participants or devise different forum mod- 
els for engaging citizens in policy deliberations on complex issues (e.g. Curato et al. 
2021). Dramaturgical analysis can assist these efforts by shifting the attention from 
design principles to the scripting, setting, staging and performances in these forums. 

It has been long acknowledged that the physical setting where deliberation takes 
place plays a crucial role in shaping the political behaviour and interaction of partici- 
pants (Elster 1998). While some settings can amplify communicative interactions and 
behaviours that deliberative democrats find valuable, others can mute them (Parkin- 
son 2012). For example, creating a seating arrangement based on multiple round 
tables versus one in which two rows of seats face each other in front of an audience 
raises different kinds of expectations from participants and from their participation 
in deliberative forums. Particular seating arrangements can also encourage informal 
interactions and produce specific behaviours (Van Maasakkers and Oh 2020). Other 
‘atmospheric qualities’ of the physical environment, such as the level of formality 
or informality can shape the dynamics of communication in deliberative forums 
(Christiansen 2015). Dramaturgical analysis can help to examine these dynamics 
and the interaction between the physical environment and forms of interactions 
(Hajer 2005a, 2005b). 

It is not only the physical environment of deliberative forums to which dramatur- 
gical analysis draws our attention. Equally important for this type of analysis is 
the discursive environment within which deliberative forums take place. A clas- 
sic example of how dramaturgical analysis can be used to study the discursive 
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environment of a deliberative forum is presented by Hajer (2005a) in his work 
on the participatory planning forum Listening to the City—a twenty-first-century 
town meeting held in New York. The purpose of this forum was to engage the 
public in a dialogue on rebuilding lower Manhattan after the terrorist attacks of 
11 September 2001. In that study, Hajer applies his dramaturgical framework to 
examine how the forum’s setting and scripting shaped its inclusivity. He finds that 
the forum was scripted narrowly for a certain audience and around a single delibera- 
tive process, which contributed to the process being abandoned by decision makers. 
Hajer argues that the organizers missed the opportunity to script a more inclu- 
sive governance storyline into the process, enabling broader and better participation 
(Hajer 2005a, 455). 

Similarly, Bernhard Wieser and Sandra Karner (2010) use dramaturgical analy- 
sis to examine how the scripting of deliberative forums can shape the roles given to 
participants and experts, and what counts as legitimate knowledge in these forums. 
Their analysis of deliberative forums on genome research in the Netherlands shows 
that the narrow scripting of these forums hindered lay participants from express- 
ing their personal experiences as legitimate considerations. Such insights are crucial 
for both scholars and practitioners of deliberative democracy; they can inform their 
efforts to make deliberative forums more effective and inclusive. Researchers inter- 
ested in using dramaturgical analysis for the empirical study of deliberative forums 
should pay particular attention to the scripting and staging of these forums, which are 
intentional and strategic activities that can enable or hinder meaningful and inclusive 
deliberation. 


Dramaturgical Analysis of Deliberation in the Public Sphere 


Dramaturgical analysis can also be used to study the performative aspects of delib- 
eration in the public sphere. Here, researchers might focus on how political leaders 
script and stage public debate, especially after a dramatic political event, such as a 
policy crisis, a natural disaster, or a terrorist attack (e.g. Hajer and Uitermark 2008; 
Hellman and Lerkkanen 2019). The way political leaders respond to such dramatic 
events can shape the discursive quality of the public sphere in significant ways. 
Researchers can use dramaturgical analysis to examine the performances of political 
leaders and the implications of these performances for public deliberation. Alter- 
natively, they might also explore how advisory bodies work to bring authoritative 
advice to the public stage (e.g. Hilgartner 2000), how social movements stage protests 
(e.g. Benford and Hunt 1992; Szerszynski et al. 2003; Barnett and Scott 2007; Morton 
2021; Cevik-Compiegne et al 2022), or how everyday citizens participate in debates 
taking place in the public sphere (e.g. Hendriks et al. 2016; Jones 2020; Ercan and 
Hendriks 2022). 

One of the major appeals of applying dramaturgical analysis to study contempo- 
rary public spheres is that it enables researchers to analyse both verbal and non-verbal 
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modes of expression and interaction. We know that public deliberation is not con- 
fined to speech and text; it also encompasses non-verbal modes of expression, such as 
visuals, sound, and presence (Mendonga et al. 2022). Actors in contemporary public 
spheres draw increasingly more on non-verbal modes of expression to display their 
arguments and take part in public deliberation (Ercan et al. 2022). Yet the methods 
to study non-verbal expression in deliberative democracy are yet to be established. 
Dramaturgical analysis offers one way of examining non-verbal communication in 
public deliberation. 

In our study of a polarized political controversy over Coal Seam Gas (CSG) min- 
ing in Australia, we turned to dramaturgy to help us make sense of non-verbal 
communication, including playful visual material circulated on social media both 
by proponents and opponents of CSG projects (Hendriks et al. 2016). Applying 
dramaturgy in online environments required making some adaptions to standard 
dramaturgical frameworks and data generation. Our analysis focused on the dra- 
maturgical dimensions of the setting, scripting, and staging of six Facebook pages 
associated with a controversy surrounding a proposed CSG project in Narrabri, in 
eastern New South Wales. For each Facebook page we analysed the organizational 
description of the page, the imagery and text on the cover and in profile images, the 
imagery and text of original and shared posts, as well as the comments, all over a 
four-month period. This analysis helped us to understand how like-minded publics 
form and perform on social media, and how they enact arguments and establish 
boundaries. 

In a different study on the same controversy, we undertook a dramaturgical anal- 
ysis of the protests enacted by a social protest group, the Knitting Nannas Against 
Gas (KNAG) (Ercan and Hendriks 2022). Members of this group meet regularly in 
public places to knit yellow and black objects such as beanies (hats), scarves, and toys 
to oppose CSG and to express care for land and communities. In this study we under- 
took a dramaturgical analysis to examine how KNAG scripts its performances, the 
central characters and audiences it employs, and the deliberative effects of these per- 
formances in the public sphere. Particular attention was paid to the props, symbols, 
and colours that KNAG uses to evoke a particular emotion or mood. Here, dramatur- 
gical analysis enabled us to explore the ways in which the colourful performances of 
KNAG challenge the dominant ideas about ‘who is affected’ by CSG, and facilitate 
different voices to enter the controversy. Through their casting of the central char- 
acter ‘nanna (wise older women), and supporting characters, for example ‘kiddies’ 
(children), KNAG is able to discursively represent previously excluded voices, such 
as future generations (Ercan and Hendriks 2021; Hendriks et al. 2020). 


Dramaturgical Analysis of Deliberative Policymaking 
Processes 


Finally, dramaturgical analysis can be used to study political communication in and 
around deliberative policymaking processes. By studying the staging and setting 
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of political communication within complex policy contexts, researchers can gain 
insights into ‘what is said, what can be said, and what can be said with influence’ 
(Hajer and Versteeg 2005, 345). Scholars interested in deliberative democratic themes 
can apply dramaturgy to illuminate different discursive and democratic aspects of 
unfolding policy sequences or events (e.g. Hendriks 2009; Visram et al. 2020). It is 
especially well-suited for studying new or emerging forms of governance where there 
might be unconventional, unforeseen, or disruptive activities (Hajer and Versteeg 
2005; Yuana et al. 2020). 

An example of applying dramaturgy to study democratic aspects of contempo- 
rary policymaking is Carolyn Hendriks’ (2009) analysis of the enactment of political 
representation in governance networks associated with energy policy reforms in 
the Netherlands. This study, which combined dramaturgical and discourse analy- 
sis (after Hajer 2005a), examined how the state and other actors staged, scripted, 
and discursively constructed representation. In the dramaturgical aspects of this 
study, Hendriks analysed a series of network arrangements that were established and 
scripted by the Dutch government as part of an energy reform programme (2004- 
2008). The dramaturgical analysis probed questions such as: how were the policy 
interactions constructed, who was active on the stage, and who was the audience? 
How were characters selected, and what acting cues were they given? How did the 
interaction and its setting produce and change meanings and power relations? The 
dramaturgical analysis showed how the Dutch energy reform programme at the time 
created a complex layering of network arrangements (or stages) upon which different 
meanings of representation were performed. The discourse analysis focused more on 
the language of representation, for example how it was articulated by relevant pol- 
icy actors in key policy documents. The combined effect of this layering, Hendriks 
(2009) argued, was the emergence of a kind of ‘democratic soup’ where actors and 
institutions enacted alternative meanings of representation that were mixed in with 
representative democracy’s emphasis on political authorization, accountability, and 
responsiveness. 


Conclusion 


Politics has long been recognized as a site of drama. It is full of intriguing char- 
acters, scripts, settings as well as staged and improvised events (Rai et al. 2021). 
These performative aspects of politics are arguably more abundant and complex 
than ever before, given the expansion of where and how modern politics is enacted 
(Theocharis and Van Deth 2018). In this chapter we have shown that deliberative 
democrats have much to gain by viewing and examining public deliberation through 
a dramaturgical lens. Dramaturgy offers scholars a useful analytical framework for 
making sense of social interactions and communication in a variety of spaces of 
public deliberation. Our examples demonstrate how dramaturgical concepts can 
be applied and adapted to examine the scripting, setting, staging, and perfor- 
mances of deliberative forums, and more broadly to study the enactment of political 


Selen A. Ercan and Carolyn M. Hendriks 329 


controversies, discourses, and democratic ideals in the public sphere and policymak- 
ing processes. 

Dramaturgical analysis is an interpretive mode of analysis. Given this, some 
of the criticism levelled against interpretive research in general (mainly from a 
positivist research perspective) also applies to dramaturgy. These include issues 
such as the ‘reliability’ of the data, ‘generalizability’ of results, and the ‘replicabil- 
ity’ of the analysis (for effective responses to these criticisms from an interpretive 
perspective, see Schwartz-Shea and Yanow 2012). The purpose of dramaturgical 
analysis is not to draw generalizations, but to facilitate ‘a view of micro-practices 
with a gaze drawn to the specific and contextual (Corrigan and Beaubien 2013, 
311; our emphasis). We contend that this contextualized perspective is crucial 
for deepening and advancing knowledge on the practice of public deliberation 
(Ercan et al. 2017). 

Sceptics within the interpretive research tradition might question the value of 
dramaturgy and ask what it offers in addition to existing qualitative methods. In 
our view, dramaturgical categories enable researchers to go beyond the dominant 
‘structure vs agency’ thinking when describing or explaining human behaviour and 
interactions. They draw our attention to the role of physical and discursive envi- 
ronments in shaping productive interactions and conversations. Especially when 
supplemented with other methods of inquiry and analysis, dramaturgy provides a 
powerful means for studying deliberative politics in action. 
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Narrative Analysis 


John Boswell 


Barely a day goes by without a media pundit, strategic adviser or public figure refer- 
ring to the importance of winning ‘the narrative’ in public debate. We are, after all, 
‘narrative animals. We find narratives effective in representing the natural way in 
which people make sense of, and argue about, complex social and political issues 
(see especially Fisher 1987; Bruner 1991). 

But if we are narrative animals at heart, what does this mean for the hopes 
of a more deliberative form of democracy? The things we typically associate with 
narrative—drama, conflict, emotion—seem at first pass to be at odds with the things 
we typically associate with deliberation—sober reflection, consensus, and rational- 
ity. A deeper consideration, however, reveals more promising potential for mutual 
accommodation. An emphasis on narrative need not entail a repudiation of deliber- 
ative norms, and more significantly still, a messy, broad systemic view of deliberative 
democracy need not entail a repudiation of drama, conflict, or emotion. The rela- 
tionship between how people do communicate (narrative) and how they ought to 
communicate (deliberation) is more ambivalent than first appears. 

In this chapter, I argue that the value of narrative analysis to assessing deliberation 
lies precisely in this ambivalence; that because narrative can be both good and bad in 
the normative sense, exploring the implications in practice through narrative analy- 
sis can provide unique insight for deliberative democrats. In fact, narrative analysis 
offers a set of tools for unpacking precisely the sort of messy, contextual communica- 
tive and political practices that deliberative democrats are beginning to grapple with 
(see Boswell et al. 2016). 

To make this argument, the chapter makes two key moves. The first is to unpack 
precisely what narrative is and what narrative analysis entails. The second looks at 
examples of narrative analysis in practice, with a particular emphasis on how appli- 
cations of this approach can help to refine contemporary debates in deliberative 
democracy. 


Pinning Down Narrative 


In the last three decades or so, there has been a surge of enthusiasm for studying nar- 
rative in political scholarship. Like most concepts across social scientific endeavour, 
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its swelling popularity has also entailed a degree of concept stretching or slippage. 
At one end of the spectrum, narrative is seen as synonymous with anecdote, an every- 
day recounting of a specific sequence of events. At the opposite end of the spectrum, 
narrative is synonymous with what is more commonly called discourse, as a broad 
ensemble of ideas and symbols that order actors’ understanding of social and political 
affairs. Discourse is usually presented as something beyond the apprehension of most 
actors themselves, and never articulated as such (that is the work of the analyst— 
see Hajer 2006). There is extensive work across the social sciences, and in politics 
and policy scholarship particularly, that theorizes about, and makes empirical use of, 
narrative in both of these senses of the term (as outlined in Boswell 2013). Some of 
it intersects with important work in deliberative democracy. For example, a focus on 
narrative as anecdote was an important element of the ‘difference democrat’ push- 
back two decades ago against an overwhelming emphasis on rational argumentation 
in deliberative democracy (see Young 1996; Polletta 1998. Likewise, a focus on nar- 
rative as discourse has been a key driver in thinking about deliberative democracy 
at the broad scale in theory (see Dryzek 1990) and in practice (see Stevenson and 
Dryzek 2014; Parry 2019). 

However, the dominant usage of narrative in empirical politics and policy schol- 
arship, which I also adopt here, sits in between these two conceptions. The sort of 
narrative I have in mind is defined as a chronological account that helps actors to 
make sense of, and communicate about, a social or political issue (Roe 1994; Stone 
2002; Fischer 2003). It is important to acknowledge that this mid-range view of narra- 
tive cannot be completely distinguished from the other two conceptions in practice. 
A narrative in this sense can be seen as an accretion of anecdotes (Ospina and Dodge 
2005) while at the same time as a ‘surface textual’ manifestation of a deeper under- 
lying discourse (Schram 2012). Conceptual distinctions in social science often have 
blurry lines in practice. The chief value in emphasizing the mid-range view of nar- 
rative is that it represents the way in which people think about and argue about the 
controversial or uncertain issues that occupy the interests of deliberative democrats. 

As this definition attests, there are clear affinities and overlaps with other 
approaches to the study of discourse used prominently in research on deliberative 
democracy. This edited volume itself contains methods focusing on the analysis of 
discourses (Chapter 20), or frames (Chapter 24), for example. So, let’s cut to the 
chase: what in particular does narrative analysis have to offer? I have elsewhere made 
this argument in more detail (see Boswell 2013), but the particular key is that any 
narrative must be enacted. 

Contra research on discourse, which tends to present discourse as somehow inde- 
pendent of agents or as enacting its own agency, an emphasis on narrative focuses 
on enactment rather than the mere text itself. Narratives exist, but not independent 
of agents (Stone 2002; Bevir and Rhodes 2003). Narratives must be brought to life 
in and for a specific context, reproduced and rearticulated by embedded political 
actors. In short, narratives require narration. But narration is also not purely strategic 
or manipulative, in the way much of the literature on framing implies. Focusing on 
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narrative and narration acknowledges that it is not possible to make sense of issues 
in some non-biased way, absent the social construction of reality. It entails emotion, 
drama, performance (see especially Boswell 2016a). 

In the context of an evolving normative account of deliberative democracy 
that stresses the importance of performativity and non-verbal communication (see 
Chapter 14 in this book), the analytical attention to enactment is apt. Narrative 
analysis offers the tools to uncover and focus attention on these wider contextual 
features. 


Doing Narrative Analysis 


This is a book largely about opening up the black box of how to study and assess 
deliberation in various contexts; so, what are those tools in the case of narrative 
analysis? I will later reference exciting work from a range of other scholars push- 
ing the frontiers of deliberative democratic scholarship. However, in talking about 
the craft of undertaking narrative analysis, it is much easier to reflect in depth on 
my own work (outlined in most depth in Boswell 2016b). This is not to say mine 
is the only way. Far from it. In fact, there is a rich literature on narrative analysis 
in political science (Turnbull 2016), psychology (Bruner 1991), communication and 
rhetoric (Fisher 1987), organizational studies (Czarniawska and Gagliardi 2003), and 
far beyond. There are many distinct variants of narrative analysis, which are ever 
adapting and evolving in response to the emergence of new techniques and tools. 
Increasingly, many analysts are attempting to measure narrative meaning systems by 
rigidly coding textual sources against core narrative elements. Some are even so bold 
as to claim to have developed a ‘science of storytelling’ that assimilates narrative to a 
positivist paradigm (Jones et al. 2014). In my view, however, an emphasis on narra- 
tive fits more naturally with an interpretive paradigm that foregrounds the beliefs and 
practices of situated actors (see especially Bevir and Rhodes 2003), and in doing so 
holds greater potential to shed light on poorly understood dynamics in deliberative 
theory and practice (see Ercan et al. 2017). 

Of course, interpretive forms of analysis are often criticized for being relatively 
opaque. So, to confront this sticky prejudice head-on, I lay out in simple, slightly 
stylized terms (it is never quite this neat in practice!) how I have used this tool and 
to what effect. 


Narrative Context 


In this interpretive vein, the first step in coming to grips with the narratives swirling 
around a complex and contested issue in public deliberation is to establish the bound- 
aries and contours of debate—where it happens, who the key protagonists are, and 
what the primary areas of conflict are. This step is almost certainly not unique to 
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narrative analysis. In fact, it probably has analogies in any in-depth research into 
public deliberation at a broad scale. Pragmatically, I found the most useful tool 
in developing this initial contextual knowledge to come in the form of ‘helicopter 
interviews’ (see also Hajer 2006; Boswell et al. 2019). 

As the name suggests, ‘helicopter interviews are designed to give you an overview. 
They occur at the outset of the project with a small handful of key informants who 
know the context very well and have the willingness and patience to help you get to 
grips with things. Ideal ‘helicopters’ are experienced journalists, retired civil servants, 
long-standing third sector representatives, or academics with substantive expertise 
on the topic of interest. In my PhD research, I conducted three helicopter interviews: 
one with an experienced health activist; one with a public health official; and one 
with a professional body representative, all of whom I contacted through personal 
and professional networks. In each case, I was able to identify key actors and settings, 
and develop a sense of how actors with different experiences perceived the debate. 
Given my familiarity with the respondents, I was also able to ask seemingly mun- 
dane questions, make and correct basic factual errors, and hone my interview skills 
in a relatively relaxed and friendly environment. Two of these ‘helicopters’ remained 
important touchstones for me throughout the project. 


Narrative Tropes 


Having established the basic context through these ‘helicopter interviews, the next 
step entails apprehending, exploring, and cementing the structural features of the dif- 
ferent narratives. Inspired especially by the work of Deborah Stone (2002), I broke 
this down into two parts: emplotment (working out the plot) and tropes (establishing 
key rhetorical features). While the former is perhaps the more important and cer- 
tainly the more time-consuming part of the analysis, the latter helps most in quickly 
sorting, sifting, and organizing the analysis. 

Put simply, political narratives tend to entail a recurring set of tropes—most 
notably a pantomime ‘cast’ of familiar characters. Like any good story, there are 
caricatured villains, victims, and heroes. Identifying this ‘cast’ of characters—and 
the different ways in which proponents of different narratives establish and por- 
tray their villains, victims, and heroes—is, in my experience, the easiest way to 
identify and separate out the competing narratives on an issue. In my PhD work 
on obesity, for instance, it was clear that some advocates saw Big Food as the 
chief villain, while others presented the food industry as a constructive ‘part of 
the solution, and others still said little about the food industry altogether (Boswell 
2016b). Taking note of these characterizations helped in the exploration stage 
because it allowed me to sort out patterns of difference and similarity in what 
would eventually become the six distinct narratives I identified in my research. The 
emphasis on dramatic characters and conflict was also a useful spur for starting 
to think about the implications for deliberative norms such as mutual respect and 
reciprocity. 
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Narratives have a beginning, middle, and end, and, as analysed in political science, 
tend to follow one of two arcs—narratives of decline (where things go from good 
to bad, and therefore need intervention) and narratives of control (where things are 
out of control until a hero steps in to impose order) (Stone 2002). The most time- 
consuming task of narrative analysis is to sift through data to track out these elements. 
It is, unmistakably, painstaking work, and, in the case of my PhD, involved thou- 
sands of pages of text and video footage of public deliberation. (The study involved 
a comparative analysis of public deliberation on tackling obesity in Australia and 
the UK between 2007 and 2013, inclusive of media coverage, legislative debate, and 
select committee deliberations, the work of specialist task forces, and administrative 
committees in both countries.) 

Here, I found interviews a useful source of corroboration and augmentation to the 
analysis of text and video. Interviews were useful for ‘narrative emplotment’ for two 
key reasons. 

First, I could ask interview participants to articulate their narrative on the issue of 
obesity and could use that to reinforce or challenge the emplotments I was devel- 
oping. Eliciting narratives from interview participants is a well-worn strategy in 
interpretive political science (see Rhodes 2011), and in practice it was deceptively 
easy. To be clear, I did not ask participants to give ‘their narrative’ or use even the 
word narrative at all. I was conscious that most of the actors I spoke to were deeply 
committed to their cause and might not recognize their deeply held convictions as 
merely ‘a narrative. Nevertheless, it proved quite simple to elicit a coherent narrative 
with little prompting via a simple interview topic guide structured in narrative form. 
I would ask: how did obesity emerge as a public policy issue? What problems does 
it present to the health service and broader society? What should we do about it and 
who needs to do it? Often, I did not even have to ask all these questions. Sometimes 
participants would launch into the full story from the outset, without any prompting 
whatsoever. The interview format is ideal for allowing participants to inhabit their 
innate ‘narrative animal: 

Second, as the research developed, I was able to sense-check my emerging findings 
with participants with whom I had already built up a good rapport. Here, I could 
talk about different ‘narratives’ without fear of offending. In fact, these participants 
reflected that they were acutely aware of the variety of perspectives at play on the 
issue, because they had to interact with rival policy actors all the time. Running my 
interpretation past them was useful for it offered something akin to a first round of 
review (see Ospina and Dodge 2005 on co-production in narrative analysis). 


Narrative Enactment 


Perhaps most important of all for scholars of deliberative democracy at the broad 
scale is an emphasis on enactment. Narratives, as I have stressed above, are per- 
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formed. The basic idea is that it is not just what people say, but how they say it. 
The emphasis is on performativity—enactment focuses on narrators (who is giv- 
ing voice to a narrative?), the stages on which they perform (what context are they 
narrating in?), and the manner in which they manage their performance (how are 
they narrating?) (see also Parkinson 2012, chapter 2 on narrating). It focuses on 
the way people dress, the tone of voice they use, the manner in which they inter- 
act with others around them, and their use of hand gestures or physical props. A 
focus on enactment therefore gives insight into deliberative practice in and across 
settings. 

An emphasis on narrative enactment can come in two forms. One is through 
experiencing the context, noting and charting differences in the way narratives 
are performed to different audiences in different settings. The key method in this 
sense is observation in the moment. This can be done either directly or with the 
help of mediated footage (television, radio, online). The other is through a ret- 
rospective ‘insiders account of how actors navigate these different settings and 
how they reflect on the experience. The main tool in this latter sense is the ethno- 
graphic interview in which the researchers try to elicit reflections on the expe- 
rience of ‘being there’ through a lengthy and reflective interview, or series of 
interviews. (For more on the distinction between direct observations and indirect 
elicitation of reflections, and the strengths and weaknesses of each approach, see 
Boswell et al. 2019) 

In my PhD research, I attempted to do a mixture of both, albeit with the balance 
skewed to the second approach. In terms of analysing enactment in context, I was 
limited pragmatically by what I had access to—some settings and events were off- 
limits, had happened in the past, or were only available through a medium (such as 
video) which does not entirely capture ‘being there’ But still it produced important 
data. I observed a notable difference, for example, in video footage of a legislative 
committee hearing on obesity in Australia. First came a group of government offi- 
cials, dressed in suits, reading from a technical report about the strain on acute health 
services. Later came a single obese woman, voice wavering, sharing her intimate expe- 
rience of the indignities of accessing services. They gave voice to the same narrative, 
but in distinctly different ways, with distinctly different implications in deliberative 
terms. 

Nonetheless, observational insights were limited, so a key purpose of the inter- 
views, then, was to get a sense of what it was like and how different set- 
tings enabled different sorts of performances. They were undertaken with an 
eye to understanding the overall consequences for public deliberation at the 
broad scale—what was included and amplified, or excluded or muted, across 
settings of debate, and what this meant in deliberative terms (see especially 
Boswell 2015, 2016c). 
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Using Narrative Analysis for Assessing Deliberation 


Having explained in practical terms what narrative analysis involves, the rest of the 
chapter focuses on how narrative analysis can be useful to the field of deliberative 
democracy at the broad scale. To be clear, I do not want to be seen as saying narrative 
has no use as a tool for understanding the micro dynamics of deliberation in a sin- 
gle setting. Hampton (2009), for instance, demonstrates the value of an emphasis on 
narrative on mini-public deliberation about environmental management. Narrative 
policy analysis is to be found in extensively tried and proven toolkit for enhancing 
and evaluating mini-public deliberation. In contrast, it is at the broader scale where 
deliberative democracy has much greater need for effective tools of analysis and eval- 
uation (Ercan et al. 2017; Bachtiger and Parkinson 2019), so I direct my attention 
there. 


Probing Flimsy ‘Consensus’ in Controversial Debate 


The place of conflict in deliberative democracy remains a point of some uncertainty 
and contention. Though long associated with—and critiqued as—the pursuit of con- 
sensus, in fact most theories of deliberative democracy have long since its early 
formulations moved beyond consensus as a goal. They recognize that conflict on 
some issues is simply ineliminable and that the appearance of consensus is as likely to 
be underpinned by problematic discursive dynamics as by ‘perfect’ deliberation (see 
Dryzek and Niemeyer 2006; Mansbridge et al. 2010). How might we come to terms 
with these dynamics at the broad scale? 

Mark Roberts’ (2018) recent analysis of the ‘communication breakdown’ sur- 
rounding the ‘Citizen Power’ initiative in Peterborough, England breaks important 
new ground. Citizen Power was a wider programme in British civil society. In the case 
of Peterborough, it took the form of an initiative to empower citizens in four specific 
policy areas: community arts, education, drugs and alcohol misuse, and environmen- 
tal concerns. Civil society groups met with Council leaders in a series of designed 
dialogues based on deliberative principles of engagement. Roberts deploys narra- 
tive policy analysis to track the trajectory of contestation about the initiative—from 
a honeymoon period of optimistic consensus in the initial, high-profile dialogues 
through to considerable antagonism and recrimination behind the scenes later on, 
largely because proponents of competing narratives about ‘Citizen Power’ simply 
‘could not understand one another. His account provides insight into how an appar- 
ent consensus can break down in the absence of sufficiently deep common ground 
across rival narratives. What is particularly interesting from a deliberative perspec- 
tive is that Roberts’ analysis probes beneath the sort of agreement often reached in 
one-off or high-profile moments of deliberation to reveal festering conflicts about 


340 Narrative Analysis 


what those abstract ideas ought to look like in practice. It provides further ammuni- 
tion for conceptual claims that deliberative democracy ought to be just as interested 
about what happens after will formation as before, in the context of messy, iterative 
processes of real-world governance (see also Boswell and Corbett 2017; Dean 2018). 


Unpacking the Dynamics of Polarization 


Deliberative theorists have been increasingly worried about the apparently inex- 
orable polarization of society, as revealed in the rise of reactionary populism and 
opposition to the ‘cosmopolitan elite’ (see discussion in Hendriks et al. 2020). These 
macro-political forces seem to render the deliberative ‘fantasy’ ever more utopian 
(see Achen and Bartels 2017). Once more, narrative analysis provides unique insights 
into these dynamics and how to counter them at a broad scale. 

A good example is in the work of Raul Lejano and Jennifer Dodge (2017). Focused 
on the impasse over climate policy in the US, and the ‘adversarial turn pushing sides 
of the debate further and further apart, Lejano and Dodge focus on a key narrative 
underpinning scepticism. They emphasize the linguistic properties and performance 
of the scepticism narrative, and show how the narrative evolves what they call ‘ide- 
ological’ properties over time. For example, they show how frequent rehearsal of the 
climate sceptic narrative in the US entails ‘saturation’ (or extreme exaggeration) of 
some of the key tropes discussed earlier. For example, in this narrative, the casting of 
proponents of progressive climate change policy as ‘liberals’ works to delegitimate 
their claims, placing them in an ideologically encamped opposition to the narra- 
tor and their assumed audience—in particular, former Vice President-cum-climate 
change campaigner Al Gore occupies the place of chief pantomime villain at the cen- 
tre of the ‘global warming hoax’. Sceptics coalesce around the narrative, and in the 
process become progressively isolated from the influence of others in debate. Taken 
as a whole, Lejano and Dodge's analysis reveals that traditional approaches to com- 
municating climate science to sceptics ignore the deeper roots of the divide, and 
so reinforce rather than mitigate conflict and paralysis. What is required to enable 
progress on the climate debate at the broad scale, they show, is a more fundamental 
attempt to bridge narratives. 


Unpacking the Dynamics of Transformation 


Narrative analysis is not just valuable in revealing pathologies in deliberative democ- 
racy at the broad scale. It can also reveal unexpected or unappreciated opportunities. 
Particularly important—in deliberative systems’ terms—is how to breach the sorts 
of impasses described in the two examples above. How do we enable meaningful 
progress and resolution of debate in messy, polarized macro terms? The established 
techniques of democratic innovation, which have been the focus of deliberative 
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democrats for much of the previous two decades, provide few pointers. They show us 
constructive transformation (understood in terms of individual preference change) 
in isolated ‘social laboratories’ but relatively little about how deliberative transfor- 
mation might occur at the broad scale. Once more, narrative analysis can provide 
fruitful answers. 

We get an important hint of this in Marcos Engelken-Jorge’s (2018) account of 
the abolition of military service in Germany and Spain. I should note here that 
Engelken-Jorge does not describe this work as ‘narrative analysis —albeit he has writ- 
ten eloquently elsewhere about the centrality of narrative to public communication 
in deliberative democracy (see Engelken-Jorge 2016). He describes it more openly as 
a focus on mechanisms apart from rational persuasion. In practice, it entails a careful 
reconstruction of the competing narratives around military service and their evolv- 
ing interaction in settings of debate over multiple decades. The emphasis is on how 
narratives emerge and evolve as they are enacted repeatedly in public deliberation. 
His analysis reveals how the abolition narratives in both countries gained ascent, not 
through any single, cathartic moment of reflection, but through a gradual, non-linear 
process of transformation and accommodation. The key mechanism was continued 
problematization of conscription in both countries—but problematization rooted in 
different reasons, underpinned by different narratives. (In Spain, for example, he 
charts how continued bouts of problematization in the public sphere are sparked 
by those concerned morally about protecting the rights of conscientious objectors 
and those concerned pragmatically about the need to professionalize the military to 
be fit for modern warfare.) He shows us how an imperfect, messy public debate in 
both countries enabled deliberative democratic effects at the broad scale: ‘namely, to 
attain mutually justified decisions, to secure the free, reasoned and informed consent 
of citizens, and to promote substantively correct decisions’ (Engelken-Jorge 2018). 


Conclusion 


Allusion to these brief vignettes reveals that narrative analysis has much to offer 
the study of deliberative democracy, especially now that the study of deliberative 
democracy has expanded its focus on institutions and mini-publics to examine the 
wider deliberative system. These studies reveal how creative analysts can shed light on 
complex and pressing questions for deliberative theory, and for democratic practice. 

Naturally, what I see as strengths, others may point to as limitations. Creative 
exploration does not exactly go hand in hand with systemic standardization. Empha- 
sis on contextual enactment may not fare well when the goal is a rigid form of 
generalizability. Is marrying these goals the next step? 

On that point, I would sound a final word of caution. Much of the so-called ‘science 
of storytelling, which attempts to link narrative insight with mainstream social sci- 
ence, ends up blunting what is interesting about narrative (for critiques, see Lejano 
2015; Dodge 2015). It reduces narrative to rigid elements and, more importantly, 
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places artificial controls on real-world political debate. In fact, in practice it often falls 
into the same ‘social laboratory’ trap that has befallen studies of deliberative practice. 
More promising, then, might be efforts to combine approaches—André Bachtiger 
and John Parkinson's (2019) recent book represents an effort to bridge ‘two worlds’ 
in this sense—which can go beyond idiography but retain an emphasis on richness 
and context. 


Further Reading 


Boswell, John. 2013. ‘Why and How Narrative Matters in Deliberative Systems. 
Political Studies 61 (3): 620-636. 

Rhodes, R. A. W. 2018. Narrative Policy Analysis: Cases in Decentred Analysis. 
Basingstoke: Springer. 


References 


Achen, Christopher, and Larry Bartels. 2017. Democracy for Realists: Why Elections do 
not Produce Responsive Government, Vol. 4. Princeton, NJ: Princeton University Press. 

Bachtiger, André, and John Parkinson. 2019. Mapping and Measuring Deliberation: 
Towards a New Deliberative Quality. Oxford: Oxford University Press. 

Bevir, Mark, and Rod Rhodes. 2003. Interpreting British Governance. New York: Rout- 
ledge. 

Boswell, John. 2013. “Why and How Narrative Matters in Deliberative Systems. Political 
Studies 61 (3): 620-636. 

Boswell, John. 2015. “Toxic Narratives in the Deliberative System: How the Ghost of 
Nanny Stalks the Obesity Debate’ Policy Studies 36 (3): 314-328. 

Boswell, John. 2016a. “The Performance of Political Narratives: How Australia and 
Britain’s “Fat Bombs” Fizzled Out. The British Journal of Politics and International 
Relations 18 (3): 724-739. 

Boswell, John. 2016b. The Real War on Obesity: Contesting Knowledge and Meaning in a 
Public Health Crisis. London: Palgrave Macmillan. 

Boswell, John. 2016c. ‘Deliberating Downstream: Countering Democratic Distortions in 
the Policy Process. Perspectives on Politics 14 (3): 724-737. 

Boswell, John, and Jack Corbett. 2017. ‘Why and How to Compare Deliberative Systems: 
European Journal of Political Research 56 (4): 801-819. 

Boswell, John, Jack Corbett, and Rod Rhodes. 2019. The Art and Craft of Comparison. 
Cambridge: Cambridge University Press. 

Boswell, John, Carolyn Hendriks, and Selen Ercan. 2016. ‘Message Received? Examining 
Transmission in Deliberative Systems: Critical Policy Studies 10 (3): 263-283. 


John Boswell 343 


Bruner, Jerome. 1991. “The Narrative Construction of Reality: Critical Inquiry 18 (1): 
1-21. 

Czarniawska, Barbara, and Gagliardi, Pasquale, eds. (2003) Narratives We Organize By. 
Philadelphia, PA: John Benjamins. 

Dean, Rikki John. 2018. ‘Counter-Governance: Citizen Participation beyond Collabora- 
tion. Politics and Governance 6 (1): 180-188. 

Dodge, Jennifer. 2015. ‘Indication and Inference: Reflections on the Challenge of Mixing 
Paradigms in the Narrative Policy Framework: Critical Policy Studies 9 (3): 361-367. 
Dryzek, John. 1990. Discursive Democracy: Politics, Policy, and Political Science. Cam- 

bridge: Cambridge University Press. 

Dryzek, John, and Simon Niemeyer. 2006. ‘Reconciling Pluralism and Consensus as 
Political Ideals: American Journal of Political Science 50 (3): 634-649. 

Engelken-Jorge, Marcos. 2016. ‘Narrative Deliberation? On Storytelling as a Necessary 
Component of Public Deliberation. Politica y Sociedad 53 (1): 79-99. 

Engelken-Jorge, Marcos. 2018. ‘Minimally Deliberative Deliberative Systems? Problema- 
tisation and the Deliberative Democratic Effects of Poorly Deliberative Communica- 
tion. Political Studies 66 (1): 137-153. 

Ercan, Selen, Carolyn Hendriks, and John Boswell. 2017. ‘Studying Public Deliberation 
after the Systemic Turn: The Crucial Role for Interpretive Research. Policy & Politics 45 
(2): 195-212. 

Fischer, Frank. 2003. Reframing Public Policy: Discursive Politics and Deliberative Prac- 
tices. Oxford: Oxford University Press. 

Fisher, Walter (1987) Human Communication as Narration: Toward a Philosophy of 
Reason, Value, and Action. Columbia, SC: South Carolina University Press. 

Hajer, Maarten. 2006. ‘Doing Discourse Analysis: Coalitions, Practices, Meaning: In 
Words Matter in Policy and Planning: Discourse Theory and Methods in the Social 
Sciences, edited by Margo van den Brink and Tamara Metze, pp. 65-74. Utrecht: 
KNAG/Nethur. 

Hampton, Greg. 2009. ‘Narrative Policy Analysis and the Integration of Public Involve- 
ment in Decision Making’ Policy Sciences 42 (3): 227-242. 

Hendriks, Carolyn, Selen Ercan, and John Boswell. 2020. Mending Democracy: Demo- 
cratic Reform in Disconnected Times. Oxford: Oxford University Press. 

Jones, M., Shanahan, E., & McBeth, M. (Eds.). (2014). The science of stories: Applications 
of the narrative policy framework in public policy analysis. Palgrave MacMillan, New 
York. 

Lejano, Raul. 2015. ‘Narrative Disenchantment. Critical Policy Studies 9 (3): 368-371. 

Lejano, Raul, and Jennifer Dodge. 2017. “The Narrative Properties of Ideology: The 
Adversarial Turn and Climate Skepticism in the USA’ Policy Sciences 50 (2): 195-215. 

Mansbridge, Jane, James Bohman, Simone Chambers, David Estlund, Andreas Føllesdal, 
Archon Fung, Cristina Lafont, Bernard Manin, and José Luis Marti. 2010. “The Place 


344 Narrative Analysis 


of Self-Interest and the Role of Power in Deliberative Democracy’. Journal of Political 
Philosophy 18 (1): 64-100. 

Ospina, Sonia, and Jennifer Dodge. 2005. ‘It’s About Time: Catching Method up to 
Meaning—the Usefulness of Narrative Inquiry in Public Administration Research. 
Public Administration Review 65 (2): 143-157. 

Parkinson, John. 2012. Democracy and Public Space: The Physical Sites of Democratic 
Performance. Oxford: Oxford University Press. 

Parry, Lucy. 2019. ‘Discourses on Foxhunting in the Public Sphere: A Q Methodological 
Study’ British Politics 14 (3): 290-310. 

Polletta, Francesca. 1998. ‘Contending Stories: Narrative in Social Movements. Qualita- 
tive Sociology 21 (4): 419-446. 

Rhodes, R. 2011. Everyday Life in British Government. Oxford: Oxford University Press. 
Roberts, Mark. 2018. ‘Communication Breakdown: Understanding the Role of Policy 
Narratives in Political Conflict and Consensus. Critical Policy Studies 12 (1): 82-102. 

Roe, Emery. 1994. Narrative Policy Analysis. Durham, NC: Duke University Press. 

Schram, Sanford. 2012. “The Deep Semiotic Structure of Deservingness: Discourse and 
Identity in Welfare Policy: In The Argumentative Turn Revisited: Public Policy as Com- 
municative Practice, edited by Frank Fischer and Herbert Gottweis, pp. 236-258. 
Durham, NC: Duke University Press. 

Stevenson, Hayley, and Dryzek, John S. 2014. Democratizing Global Climate Governance. 
Cambridge: Cambridge University Press. 

Stone, Deborah. 2002. Policy Paradox: The Art of Political Decision Making. New York: 
W.W. Norton & Co. 

Turnbull, Nick. 2016. ‘Narrative and Interpretive Theory. In Handbook on Theories of 
Governance, edited by Chris Ansell and Jacob Torfing, pp. 380-390. Cheltenham: 
Edward Elgar Publishing. 

Young, Iris Marion. 1996. ‘Communication and the Other: Beyond Deliberative Democ- 
racy, In Democracy and Difference: Contesting Boundaries of the Political, edited by 
Seyla Benhabib, pp. 120-136. Princeton, NJ: Princeton University Press. 


24 
Frame Analysis 


Ricardo F. Mendonca and Paula G. Simões 


Deliberative democracy involves exchanges of discourses across time and space. To 
understand how discourses emerge, how they challenge each other, how they are 
revised and transformed, and how they pervade the way social actors ascribe mean- 
ing to these exchanges is a challenge for any scholar of deliberative democracy. In 
this chapter, we introduce frame analysis as one way of addressing this challenge. 

In simple terms, frames answer the question: ‘what is happening here? Humans 
need this basic definition in order to make sense of what they are doing, what is 
expected from them, and how they may act within particular social interactions. 
Frames are interpretive packages (Gamson 1992), which articulate meanings, throw- 
ing light on particular elements of reality while obscuring others. They are enacted 
and updated in human interaction. 

Frame analysis is a method that describes how frames are produced, mobilized, 
and interact with other frames. It reveals how individuals position themselves in 
relation to existing frames and identifies frames’ consequences in specific contexts. 
The epistemological assumptions of frame analysis are aligned with core elements of 
deliberative democracy. First, it is grounded on the premise that meaning is created, 
crystalized, or displaced intersubjectively. It, therefore, challenges methodological 
individualism and echoes the deliberative attention to interaction, mutuality, cooper- 
ation, and discursive exchange (Mendonga and Santos 2009). Second, frame analysis 
conceives of discourses as constitutive of reality, and not merely as descriptions of 
it. As van Gorp argues (2007, 73) ‘the purpose of a frame analysis is to assess not so 
much the impact of loose elements in a text but the impact of the implicitly present 
cultural phenomena conveyed by all these elements as a whole and to relate them to 
the dynamic processes in which social reality is constructed: Third, frame analysis 
places context at the heart of meaning-making, thus shedding light on the impor- 
tance of situating discourses in the context in which they were expressed. The current 
attempt of deliberative democrats to understand how different communicative con- 
texts may offer diverse contributions to deliberative processes is grounded on the 
same premise that discourse is not only content, but a contextualized production 
(Goodin 2008; Mansbridge et al. 2012). 

Having established the clear connection between deliberative democracy and 
frame analysis, the question now is how exactly frame analysis can be used in the 
study of deliberative democracy. Our answer to this question will be presented in 
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two sections. The first section presents three ways of using frame analysis while the 
second section provides illustrative case studies. 


Frame Analysis: Varieties and Potentials for Deliberative 
Scholarship 


Frame analysis can be conducted in various ways, leading Entman (1993, 52) to 
describe it as a ‘fractured paradigm: Mendonga and Simoes (2012) identify three 
ways of operationalizing frame analysis. By presenting these three ways, we do not 
mean to provide a rigid typology, but rather to point out the different emphases in 
the use of frame analysis. In what follows, we introduce each of these three perspec- 
tives, provide a brief description of how to employ each one, and argue how it can be 
useful for deliberative democrats. 


Investigating Sequences of Interaction 


The first way to conduct frame analysis is to reconstruct sequences of interaction. 
The aim is to reveal how individuals employ and displace certain interpretive pack- 
ages to make sense of what is going on. This mode of operationalization looks at how 
different contexts shape the relationships established therein, and how actors oper- 
ate within these situations. It grasps their fragility by looking at observable ruptures 
along interactions and at the following efforts to generate new alignments and mutual 
adaptations. 

The analysis begins with the identification of interactive strips, which are the unit 
of analysis. Interactive strips are sequences of interaction with turns from different 
actors. When studying a deliberative mini-public, for example, an interactive strip 
would be a specific excerpt of the relationships therein established. In other words, 
it is a conversational fragment. The notion of sequence is relevant here because it 
draws attention to the idea that actions are intertwined, with each step affecting the 
development of a given interaction. The focus is hence on dynamics of interactions 
and not on isolated utterances. 

By reconstructing fragments of broader conversations, frame analysis investigates 
how discourses are defined and redefined as actors engage in interaction. The follow- 
ing are useful questions when conducting frame analysis: What is happening there? 
Who are the ones taking part in that situation and how do they position themselves? 
How do they make sense of what is going on? Are there conflicting attempts to define 
what is going on? How do actors deal with alternative attempts to define the situa- 
tion? What are the rules and grammar that govern the interaction? What happens 
when interaction is broken? 

Studying the relationships pervading interactive sequences, scholars working from 
this perspective may uncover struggles over framing and over the right to make 
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oneself heard in given contexts. Reconstructing sequences of interaction enables 
deliberative democratic scholars to comprehend how actors position themselves 
intersubjectively for the construction of interpretive patterns to make sense of real- 
ity. This interactive approach offers a qualitative way to deal with reason-giving and 
reciprocity. It sheds light on how they operate in specific contexts and in specific 
language games. 

How frames operate in specific contexts is a central part of this analysis. Discursive 
exchanges must be thought of within the contexts in which they happen. At the heart 
of meaning production are the addressees of messages, the devices used to produce 
and to receive these messages, the communicative features of the interaction, and the 
broader social environment in which particular interactions are nested. The inher- 
ently situated lenses provided by this approach foster an attention to the different 
shapes that discursive exchanges can assume. At this point, it is relevant to remember 
that the starting point for the deliberative systems approach is that different venues 
have different features and that the articulation of arenas with diverse qualities may 
be good for deliberation (Goodin 2008; Dryzek 2012; Mansbridge et al. 2012; Elstub 
et al. 2018). Investigating deliberation through a systemic lens entails reconstructing 
discursive situations so as to understand how utterances are produced, to whom they 
are addressed, how they circulate, and what the rules are that shape interactions in 
specific settings. By shedding light on the centrality of situation and context, frame 
analysis can help deliberative scholars to make sense of the role played by different 
venues in a deliberative system. Different discursive arenas are sites of diverse com- 
municative situations. They create the conditions for certain forms of expression, 
while inhibiting others. Through frame analysis, deliberative scholars can investi- 
gate sequences of interaction in order to understand the affordances that different 
discursive situations provide for public debates. 


Mapping Public Clashes of Discourses 


Frame analysis is also an approach to mapping debates around controversial issues. 
At the kernel of this approach lies an attempt to understand the way specific dis- 
courses frame political topics, such as war, climate change, nuclear energy, abortion, 
or elections (Gitlin 1980; Gamson 1992; Ferree et al. 2002), and how social actors 
mobilize frames in public battles over interpretations (Cefai 2007; Snow et al. 1986; 
McAdam 1996). Scholars grounded on this approach often seek to comprehend how 
the content of discourses establishes a context of meaning. The interpretive route 
to do this, broadly speaking, follows this sequence: (1) define a problem; (2) offer 
a causal interpretation; (3) evaluate it in moral terms; and (4) provide a treatment 
or recommendation (Entman 1993, 52). Frames are thus employed as analytical 
mechanisms to develop content and/or discursive analysis. 

This mode of frame analysis may involve an inductive procedure, through which 
the scholar navigates a selected public debate, identifying: (1) the arenas in which it 
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happens; (2) the actors involved in building that discursive field; and (3) the recurrent 
patterns of interpretation mobilized by these actors. Van Gorp (2007) suggests that 
this reconstruction of patterns of interpretation (i.e. frames) should be attentive to 
three things: (1) the framing devices: which words, metaphors, exemplary cases, visu- 
als are used; (2) the reasoning devices: which justifications, causes, consequences, and 
equivalences are presented; and (3) the implicit cultural phenomenon: what does this 
frame reveal about broader patterns of interpretation and social values? Each pattern 
(or frame) must be transformed into a narrative as a way to define a given situation 
or problem. 

Content or Discourse Analysis that employs frames from this perspective can also, 
nevertheless, operate in a deductive way, starting with pre-established categories that 
are used to code the empirical material. These categories can be derived from theoret- 
ical work or other empirical investigations. The basic difference is that the researcher 
does not build the frames from the data analysed, but uses categories developed in 
previous investigations. For example, one could draw on feminist theories to identify 
frames used by different actors in a given policy discussion about gender-based vio- 
lence. Applying deductive frame analysis can be useful for comparative enterprises, 
and it is important to employ adequate procedures to assure inter-coder reliability in 
this application. The unit of analysis varies in different studies, ranging from single 
sentences, to utterances, and even entire texts. This decision depends on the focus of 
the study and the empirical data available. 

After identifying frames (either inductively or deductively), the researcher may 
use them to answer a variety of research questions. Mapping frames can help to 
understand, for instance, the diachronic transformation of a given debate, through 
an analysis of the flows, overlaps, and displacements of interpretive packages in the 
public sphere. It can also help in identifying the factors that explain the strength or 
weakness of different frames in diverse contexts. Mapping frames may also be useful 
when seeking to diagnose whether a debate is focused or dispersed, to identify net- 
works of actors operating within the same interpretive pattern, and to comprehend 
how the addresses of discourses are relevant in word selection and metaphor usage. 
These are just a few of the possibilities afforded by this method. 

All these possibilities are relevant for deliberative scholarship, especially for those 
investigating what goes on in the public sphere. Macro deliberative scholars and those 
working within a systemic perspective seek to identify the broad discursive processes 
cutting across time and space. They are concerned with comprehensive discursive 
dynamics, looking at public opinion formation, cultural interpretive changes, or the 
capacity of debates to exert influence on decision-making processes. In offering tools 
to map wider discursive processes, frame analysis helps to track discourses travelling 
through time and space, which is essential for scholars who investigate connections 
between arenas or transmission processes within systems (Bachtiger and Parkinson 
2019; Boswell et al. 2016; Mendonca 2016). By focusing on frames, instead of par- 
ticular utterances, deliberative scholars can investigate public clashes of discourses, 
understanding how specific interpretations of a given problem have been developed. 
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Interestingly, therefore, the attention to discursive situations fostered by frame anal- 
ysis does not imply the construction of a micro approach, but can help in developing 
macro and situated approaches. 


Identifying Framing Effects 


Lastly, frame analysis is also used to investigate framing effects, which happen ‘when 
(often small) changes in the presentation of an issue or an event produce (some- 
times large) changes of opinior (Chong and Druckman 2007, 104). Applying frame 
analysis from this perspective involves conducting experiments to test the effects of 
specific discursive formulations or the consequences of media devices and cultural 
products. Surveys are also frequently used to assess the impact of certain discourses 
in public opinion and to investigate the role that different factors play in agenda- 
setting (Weaver 2007). The susceptibility of persons to frames is measured and there 
is a search for factors that may limit the impact of framing (Druckman and Nelson 
2003). In this approach, frames are often taken as a strategic discursive bias to drive 
audiences in a given direction (Reese 2007). 

Deliberative scholars can benefit from the study of framing effects in many ways. 
It can reveal, for instance, the consequences of biased informative materials in expe- 
riences of democratic innovation or of non-plural debates in the public sphere. As 
discussed below, there is also a vein of the literature seeking to argue how delibera- 
tion might control framing effects and reduce its impacts. The use of frame analysis 
to look at discourse effects is also relevant to deliberative democrats because it is 
alert to the interplay between agency and culture in the construction of discourses. 
Frame analysis is well suited to investigating how discursive construction involves a 
series of choices and strategies that affect the way audiences deal with discourses. 
Frame analysis realizes, however, that meanings always emerge intersubjectively, 
and cannot be determined by individuals who formulate them. In addition, it also 
acknowledges that the choices pervading discursive construction are made within a 
net of cultural meanings, linguistic grammars, and taken-for-granted elements. It is 
this interplay between agency/strategy and society/culture that allows a rich analysis. 
Through frame analysis, deliberative scholars can investigate the agency of delib- 
erators in political debates and also grasp the context of this agency. Actors make 
choices (about words, metaphors, definitions, arguments) within contexts, aiming to 
promote meanings and arguments that can be comprehended by their addressees. 


Frame Analysis and Deliberative Democracy 


There are many studies that employ frame analysis to investigate public discussion. 
Even if some of these studies are not explicitly working under a deliberative label, they 
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offer paths and findings that can deepen our knowledge of how deliberation works. 
One example is the work that Ferree et al. (2002) devoted to comparing public discus- 
sion about abortion in Germany and in the US. Frame analysis was employed to map 
these broad discursive scenes. The authors tracked the strength of different interpre- 
tive packages (i.e. frames) over the years, their internal changes, and their clashes 
and transformations. Sensitive to context, the investigation reconstructed the role 
of agents, key historical events, and culture in shaping abortion discourse through- 
out time. Although not explicitly deliberative, these studies deal with some of the 
key questions raised by deliberative scholars. Mapping public discourses and under- 
standing their influence over public opinion, cultural change, and decision-making 
is one of the main goals of the field. 

Other illustrations of investigations using the notion of frames that are not explic- 
itly based on deliberation, but which offer significant contributions to the field, are 
found in the studies of rhetoric. Breton (2003), for instance, suggests that the inves- 
tigation of argumentation starts by identifying the frames that create the context for 
the understanding of specific argumentative bonds. And Kuypers’ (2009) rhetorical 
criticism places frame analysis at the heart of the comprehension of communication, 
mobilizing it to investigate how problems are defined and how causes and solutions 
are discursively built. Such an approach has proven to be particularly relevant for 
those working with Critical Policy Analysis, where the links with deliberative democ- 
racy become clearer. Attempting to study public policies as communicative practices, 
the argumentative turn led by Fischer and Forester (1993) advocated the need to 
investigate how problems were defined in the discussions around a policy solution. 
Dekker (2017), Braun (2016), and Rein and Schön (1996) are examples of the use of 
frame analysis in the study of policy controversies. 

There is also a growing number of studies that use (or discuss) frames with an 
explicit reference to deliberation. The concern with framing effects seems to be 
the predominant agenda amongst deliberative scholars referring to frame analy- 
sis. Bohman (2007) fears that framing effects may restrict complex debate, while 
Barisione (2012) proposes a Deliberative Frame Analysis to identify the conditions 
that would avoid the dangers of framing effects. Many have pointed out, however, that 
deliberation itself may work as an antidote against framing effects, since the exposure 
to different perspectives may challenge the strength of single frames (Druckman and 
Nelson 2003; Druckman 2004; Barisione 2012). 

A more ambivalent conception of framing pervades recent investigations on the 
topic. Calvert and Warren (2014) acknowledge, for instance, the epistemic and eth- 
ical dangers of framing effects, which could weaken individual judgements. But 
they note that framing is inherent to communication and consequently seek to 
identify specific forms of framing that hurt deliberation. Framing effects are dan- 
gerous, according to them, when we have dominant (unchallenged) frames, when 
they have polarizing effects, and when they are group-based. For to the authors, 
mini-publics can be designed to avoid these problematic frames and this is relevant 
for the enhancement of broader systemic deliberation. Leeper and Slothuus (2018) 


Ricardo F. Mendonça and Paula G. Simões 351 


also advance a complex approach to the topic. While acknowledging the dangers of 
non-deliberative followership nurtured by frames, they highlight the importance of 
cognitive shortcuts provided by political elites through frames. 

Despite its importance, the focus on framing effects is not the only possible avenue 
to be pursued by deliberative scholars willing to work with frame analysis. As previ- 
ously argued, frame analysis can help deliberative scholars in diverse ways. Some have 
studied discursive processes to identify how certain frames may promote deliberative 
values such as inclusiveness and civility (Rinke et al. 2013; Sarmento and Mendonça 
2016). Others have employed it to comprehend the enactment of power relations in 
participatory experiments (e.g. Blue and Dale 2016).* Many scholars have used it to 
map public deliberation in diverse arenas, including the media, parliaments, and par- 
ticipatory experiments. Simon and Xenos (2000), for instance, studied media frames 
in relation to a labour strike in the US and the actual consequences of these frames 
in political developments. Blue (2016) investigated public deliberation about climate 
change, pointing out that the monopoly of scientific lenses may hinder the emer- 
gence of the plurality of frames needed to face the issue in all its complexity. Vimieiro 
and Maia (2011) studied processes of social learning and cultural change, by trac- 
ing frames about disability over a period of five decades. Ercan (2015) investigated 
the competing frames about honour killings in Germany, studying parliamentary 
discussions and the broad public debate. All these studies, amongst others, demon- 
strate the potential of frame analysis to map broader discursive processes and to 
foster comprehension about how interpretations are built and displaced in situated 
interactions. 

One of the authors of this chapter has also made extensive use of frame analysis 
in a series of studies about deliberation. Initially, Mendonça (2009) employed frame 
analysis in a study about the struggles of people affected by leprosy in Brazil, seeking 
to comprehend if (and how) discussions in different arenas attempted to promote 
the thematization of a public problem. The arenas under study were everyday con- 
versations in former leprosy colonies, a social movement newsletter, and the mass 
media. They were understood as different contexts of interaction (or loci for interac- 
tions), whose features, grammars, and rules affected the discourses uttered. Through 
a frame analysis, this research found that the social movement's discourse was aligned 
with the way the media reported the disease, missing many of the complexities and 
feelings of injustice pervading everyday conversations in former colonies. 

Mendonça and Santos (2009) and Mendonça and collaborators (2014) have used 
frame analysis to advance a broad conception of reciprocity that is not based on an 
individual-to-individual relationship. They claimed that discursive reciprocity should 


* It is worth highlighting the overlaps between this agenda and the growing literature on Dramaturgical 
Analysis, due to its attention to the reconstruction of the contextual dynamics pervading meaning-making 
processes (Hendriks et al. 2016; Escobar 2015; Hajer 2005). Drawing from Goffman (1986) and developing 
a specific trend in the debates over interpretive methods, this literature uses the notion of frames to make 
sense of how actors engage in interactive processes to define given situations. This literature will not be 
deepened here as it covered in Chapter 22 of this book, authored by Selen Ercan and Carolyn Hendriks. 
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not be limited to an interpersonal level, as it happens through the public clash of dis- 
courses. Investigating a referendum about the commercialization of firearms and a 
public consultation about political reform in Brazil, these articles mapped the public 
clash of frames. Furthermore, they argued that discursive reciprocity could involve 
intra-frame and extra-frame arguments (Mendonça et al. 2014). The former refers 
to situations in which diverse positions are advocated within the same frame, while 
the latter refers to situations in which a frame is mobilized to displace other frames, 
thus moving the debate in different directions. In the case of political reform, for 
instance, discussions about the electoral system were driven by a specific frame (indi- 
vidual/party) with opposing arguments about the benefits of the maintenance of an 
open-list proportional system or a change either to closed lists or to majoritarian sys- 
tems. This was quite a contrast to the discussion about campaign funding, in which 
there was a clash between two different frames: one focused on equity and fair com- 
petition, and another focused on public expenditure. We have, therefore, mapped 
two different dynamics of discursive reciprocity: argumentative exchanges orbiting 
one frame (intra-frame reciprocity) and argumentative exchanges with clashes of 
interpretive frames (extra-frame reciprocity) (Mendonça et al. 2014). 

This range of studies points to the considerable potential for the use of frame anal- 
ysis to solve puzzles that are of interest to deliberative scholars. This does not mean 
that the method should be conceived of as the technique for the study of deliberation. 
Frame analysis does not seem well suited, for instance, to assess the deliberativeness 
of specific arenas or to evaluate the quality of arguments. Nonetheless, it can help to 
address some of the central questions in the field of deliberation. 


Conclusion 


This chapter argued that frame analysis can play an important role in the study of 
deliberative democracy. It answers questions related to discursive dynamics, pointing 
to the emergence and the transformation of interpretive perspectives. As a method, it 
can be effectively applied to the study of deliberative politics for the following reasons: 
(1) it is context-sensitive; (2) it challenges methodological individualism; (3) it allows 
the reconstruction of broader discursive processes that are spread over time and 
space; (4) it is attentive to the interplay between agency/strategy and society/culture; 
and (5) it analyses the potential dangers of some framing effects to deliberation. 
Frame analysis emphasizes the intersubjective and cultural dimension of discourses 
and can be fruitful in studies that deploy a deliberative lens. 
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25 
Talk-based Analysis 


Paromita Sanyal 


Deliberative democracy is a normative political ideal that was long considered the 
reserve of literate and egalitarian enclaves of the Global North. The fulcrum of 
this form of democracy rests on consensus-building deliberations amongst open- 
minded and persuasive citizens who can sway others with the power of their reasoned 
arguments and are willing to be swayed by the arguments of others. Early propo- 
nents of this political philosophy assumed an ideal society where all citizens were 
endowed with equal communicative privileges and prowess. Much of the early empir- 
ical research focused on town hall meetings in New England, USA, where concerns 
about social stratification were suspended (Mansbridge 1980). Deliberative forms 
of governance have since diffused from ideal enclaves in high-income countries to 
unequal communities in middle- and low-income countries. In recent scholarship 
there is broader recognition of deliberation-based governance institutions and prac- 
tices that have multiplied in the Global South (Curato et al. 2017; Dryzek 2000; Heller 
and Rao 2015). Some of these institutions are rooted in historical antecedents and 
have been mandated countrywide, like Indian gram sabhas (village assemblies), while 
others are newer innovations. 

At this juncture of its global diffusion, we need to diversify the questions we ask 
about deliberative democracy. It is not enough to pursue classic concerns such as 
evaluating the quality of public opinions and collective decisions against the yard- 
stick of ends-and-means discussions and consensus building. We also need to worry 
about how social inequalities and social policies affect the process of deliberation and 
citizens’ deliberative capacities. We need to examine a new set of concerns: how delib- 
erative forums introduce the scope for new kinds of communication amongst citizens 
and between citizens and the state; how citizens perform their community mem- 
bership or citizenship in these deliberative forums; and how civic bodies and states 
perform their authority and facilitate or obstruct deliberative governance efforts. 
There are multiple dimensions of the institutions and practices of deliberative democ- 
racy, and these have political as well as sociological ramifications. For studying each 
of these, we need multiple methodologies. This chapter proposes one such method, 
a qualitative method, which engages in talk-based analysis. In the rest of the chapter, 
the method is described, and its distinctive advantages and limitations are discussed. 

Analysing talk in deliberative forums is full of potential. It can allow us to char- 
acterize citizens’ performances and state/organizational enactments and to form a 
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qualitative assessment of deliberations. The core of the analysis is an interpretive 
examination of verbal discourse (discussions and speeches). This includes demands 
that are articulated, arguments and counterarguments that are formulated, con- 
cerns and critiques that are voiced, and rhetoric presented for public and political 
consumption. So, rather than solely following the evolution and culmination of deci- 
sions, this method focuses our attention on who talks in deliberative forums, what 
they say, and how they speak and express themselves. In contexts of proximate 
inequality, where citizens gathering for deliberations are objectively and symboli- 
cally unequal in resources and recognition, it is important to focus on how citizens 
articulate their demands and decisions, respond to each other, and hold the state 
or other agencies accountable. In a fundamental way, then, this method focuses on 
understanding deliberative democracy from a sociological perspective. It does not 
discount communicative exchanges in the public sphere that fail to meet the rigor- 
ous normative standards of ‘deliberation. Rather, it is an analytical method in which 
civic deliberations are considered analogous to a drama in which people participate 
and perform. 

Thus, the proposed method can be used to examine cool and considered deliber- 
ations as well as deviations from this norm that include highly charged emotional 
renderings, narratives, competing claims, derisive critiques, and also chaos and con- 
flict. By analysing speech (what people talk about and how), this method also allows 
for a systematic analysis of how social stratification and state policy act as moderators 
and mediators of deliberation. Therefore, the unique advantage of talk-based analysis 
is that it allows one to (a) characterize the variations of voice, styles of articulation, 
and discursive performances, and (b) identify the underlying patterns of power and 
privation amongst citizens and the state that may be associated with deliberative par- 
ticipation. In the broadest sense, this method is a stepping-stone that can be used to 
address key higher-order questions. Is it possible to have good deliberations in con- 
texts of inequality? Do disadvantaged citizens have the same competency as their 
advantaged peers to engage in informed deliberations? What factors influence citi- 
zens competency or capacity for good deliberations? And what can institutions like 
the state do on a large scale to improve deliberations and reach for their normative 
ideal? 


Data Generation: Sampling, Recording, and Transcribing 
Deliberative Assemblies 


Public assemblies tasked with deliberations on community governance can vary in 
size, regularity, and constitutionality. Even within the same geopolitical unit, the local 
contexts that host public deliberations may vary in background conditions. These 
features can be leveraged in the sampling methodology to maximize our knowledge 
of how social stratification and policy may be related to differences in deliberation. 
This section begins with a discussion of the appropriate units of analysis and the 
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sampling strategy, with illustrative examples cited from our own research (Sanyal 
and Rao 2019). 

The unit of analysis can be any deliberative forum or public meeting, where people 
engage verbally through discussion and debate. For deliberation scholars, commu- 
nity governance and development are core concerns. So, deliberative institutions 
centred on these concerns are prime units of analyses. These could be civic meet- 
ings in the public sphere organized by communities, by the local government or state 
agencies, or by non-governmental organizations and development agencies. 

Public meetings can be sampled following a variety of strategies informed by the 
research question, including ones guided by natural experiments. In our study of 
Indian village assemblies (Sanyal and Rao 2019), we chose our village assemblies 
following two sampling strategies for comparative analysis. We sampled assemblies 
from villages that were in language-matched blocks (which we take as a proxy for 
similarities in discursive cultures) that fell across modern state lines but historically 
belonged to the same larger administrative unit in pre-colonial India. By comparing 
these across-state village assemblies that were presumably culturally similar, we were 
able to examine the local state and political regime’s role in shaping deliberations 
(through things like information-sharing with citizens, crafting an agenda, requiring 
public officials to be present). We also sampled village assemblies within the same 
state and district but with varying levels of literacy. By comparing these within-state 
village assemblies, we were able to explore the patterns of deliberation and citizens’ 
deliberative capacities associated with high, medium, or low literacy levels. 

For talk-based analysis, deliberations need to be recorded and transcribed. Trans- 
lation to English will be necessary for transcripts of meetings occurring in other 
languages, if the research is intended to reach an English-speaking audience. Tran- 
scription and translation must be done with the utmost care since the substance of 
analysis is what people say and how they say it. Participants often use idioms and 
allegories and speak with sarcasm, all of which must be captured rather than lost 
in translation. For analysing social stratification in deliberations, transcripts need 
to include information on the relevant group and the gender identity of the speak- 
ers. In our study, each transcript was accompanied by corresponding information on 
the gender, caste identity, and the social position of the speakers (ordinary villager, 
schoolteacher or principal, club leader, women’s self-help group leader or member, 
bureaucrat, etc.). 


Qualitative Analysis: Coding and Interpreting Talk 
and Performances 


The talk-based analysis proposed here draws inspiration from a few key sources. 
Mansbridge (1998), one of the early proponents of deliberative democracy, recog- 
nized the performative aspect of deliberation. In her feminist critique of deliberation, 
she acknowledged early on that deliberation may turn into theatre, and argued: 
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[A] history of relative silence makes women political actors more likely to 
understand that when deliberation turns into theatre, it leaves out many who are 
not, by nature or training, actors. When deliberation turns into a demonstration of 
logic, it leaves out many who cannot work their emotionally felt needs into a neat 
equation ... Many shy men are quiet, but the equivalent percentage of shy women 
is increased by learning silence as appropriate to their gender. 

(Mansbridge 1998, 152) 


In this critique, there is an implicit call to analyse communication styles and per- 
formances as a way to dissect deliberation. But, deliberative democracy as a field 
has largely veered away from in-depth analysis of this sort. There are rare excep- 
tions (Mansbridge 1999; Dryzek 2010). Research on civic participation in Sociology, 
however, has directed increasing attention at citizens’ communication styles and nar- 
rative strategies. Eliasoph (1996, 2003) in her study of American civic participation 
proposed ‘talk-centered’ analysis. The talk-based analysis method proposed here is 
directly inspired by, and builds upon, this original version. To this has been added a 
focus on analysing deliberative participation from a performative angle. This take is 
inspired by Alexander’s (2006, 2011) focus on social performances in the civil sphere 
from a cultural theory perspective and his interpretive analysis of political power 
and performances based on discursive materials such as political speeches. Other 
contemporary sociological research on civic participation and deliberation has also 
focused on analysing narrative strategies and discursive styles (Edgell et al. 2016; 
Polletta and Lee 2006). This section outlines ideas for the kinds of analysis that can 
be conducted using this approach and what we might learn from them. 


What Is Spoken About and Who Speaks? 


One analytic strategy can be to develop inductive codes for the issues that come up for 
discussion. For example, inductive codes can be developed for public concerns and 
collective demands that are raised for discussion, and private/individual issues and 
demands that are voiced. As a next step, analysis can be aimed at tracing the source of 
these public and private issues raised. Was an issue introduced by a top-down agenda 
or by citizens/deliberation participants; by a woman or a man; by a person endowed 
with some authority in the community or by a person representing a group or by a 
lay participant; by a person with some material or symbolic advantage or by a person 
marked by disadvantage? Further, which issues are taken up for collective discussion 
and which fade away without discussion can also be subject to systematic analysis. 
One can also identify the underlying reasons as to why one issue is taken up for public 
discussion while another is not. 

This type of analysis can be used (a) to evaluate the extent of public-mindedness 
of community or governance deliberations, and (b) to understand if that public- 
mindedness is a feature driven by action from above or by a surge from below. It 
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can also be a powerful tool for revealing how social stratification may be associated 
with patterns of who speaks, what is spoken about, and what is responded to, and 
what results emerge from a general discussion. 


How People Speak 


This method is uniquely positioned to analyse and interpret the types of speech 
citizens use and the narrative strategies they employ in deliberative exercises. 
Researchers could inductively identify how people articulate their arguments and 
frame their demands and how they convince and critique others. Coding each of 
these styles and strategies with appropriate descriptive labels would generate a list of 
codes that can then be systematically applied to code the entire corpus of data. 
Communicative styles are embodiments of citizens’ sense of power and entitle- 
ment or the lack of it. Through interpretive analysis, we can classify communicative 
styles by the sense of power or powerlessness they embody. This type of analysis can 
reveal the power dynamics in deliberative settings. Viewing communication through 
a performative lens, we can then interpretively associate the communication content 
and style of a speaker with a performance they are carrying out in the deliberative 
arena. Therefore, the variety of communicative styles identified in the data could be 
assigned to a repertoire of performances that are on display in the deliberative forums 
studied. Where deliberative exercises bring citizens face to face with elected leaders 
and frontline bureaucrats, the communicative styles and performances used by indi- 
viduals may even give us a pulse on the relationship between citizens and the local 
state. Through this type of analysis, then, we can arrive at a nuanced understanding of 
how deliberative spaces become sites for citizens’ performances and state enactments. 


Illustrative Examples 


Examples from our work (Sanyal and Rao 2019; Sanyal et al. 2019) are presented here 
for illustrative purposes. The first project was a study of village assemblies across 
four Southern Indian states. Our goal was to analyse if the states’ political regimes 
and the village literacy levels made a difference in deliberations. We started with an 
inductive coding strategy that began with reading and rereading the transcripts with 
an eye to identifying patterns—the kinds of issues citizens raised, the types of citi- 
zens who were more or less likely to speak or raise certain issues, and a central focus 
on the narrative strategies that citizens used. After spending a significant amount of 
time reading our data, we began writing extensive memos listing all the patterns we 
could identify. One note about collaborative projects is warranted here. Like ours, 
many projects may be collaborative with multiple investigators. For these projects it 
is important for each investigator involved in the analysis to read the transcripts and 
establish a schedule of routine deliberations for collectively discussing the patterns 
they are each able to identify. Through this process a commonly agreed upon set 
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of codes or patterns can be identified. Then will begin the task of coding all the 
transcripts following this set of codes or patterns. Qualitative coding is a laborious 
manual process that must be carried out meticulously, whether researchers are using 
traditional memo-writing or using software, such as NVivo, Atlas.ti, or others. 

In our project, this immersive inductive process led us to identify a set of themes 
and patterns (discussed below) that consistently appeared in the transcripts. Politi- 
cal leaders and public officials were prone to schooling citizens in deliberation; they 
encouraged citizens to cooperate with the state and cooperate mutually to undertake 
collective action; and they engaged in instilling in citizens a sense of civic and fiscal 
responsibility. Later, we went on to group these codes under the master code ‘supply 
of governance’ In a similar fashion, we noticed citizens holding the state accountable 
for implementing various public works projects, addressing public goods deficien- 
cies, and raising larger issues affecting quality of life. We grouped these themes under 
the master code of ‘demand for governance. We also noticed competition over public 
goods and competition for personal goods (claims made on the basis of caste status 
and economic status) as well as claims for dignity made by disadvantaged citizens 
lacking in social recognition. We also found a patterned difference in the frequen- 
cies with which these themes appeared in the assemblies. The biggest differences were 
across state lines, showing the influence of state regimes on these deliberative forums. 

As we continued reading our transcripts through the comparative framework, we 
noticed a variety of articulation styles used by citizens and patterned state-wise differ- 
ences in how citizens spoke in these assemblies. In some assemblies, citizens appeared 
from their speech to be desperately seeking the government's attention and pleading 
for help. In other assemblies, citizens were scathingly critical of government inaction 
and neglect, and vociferously denounced political leaders for their failings. In a few 
other places, citizens were rebellious, prone to being unruly and creating an anarchic 
atmosphere. In some places, a lot of factual information was disseminated by the 
agents of the state and discussed by the citizens, whereas in other assemblies, hardly 
any facts and figures relating to budgets and public works were discussed. In assem- 
blies in some states, citizens were able to marshal a great deal of financial information 
and factual knowledge while framing their demands and complaints and holding the 
state accountable, while in other states, citizens were not knowledgeable of budgetary 
information or aware of the progress of public works projects undertaken by the local 
government. In addition to paying attention to the content and framing of speeches, 
we were also attentive to the emotional character of speeches by citizens, political 
leaders, and bureaucrats. These differences were most apparent across state lines and 
modest across literacy differences. 

At the end of this phase of analysis, we were able to identify clear differences in the 
content, style, and strategy of citizens’ speeches. We could also distinguish between 
the moral or pragmatic character and the facilitative or deflective strategy of speeches 
by elected leaders and public officials. Lastly, we proceeded to characterize citizens’ 
performances and state enactments by developing conceptual labellings. Table 25.1 
lists our interpretive labellings. 
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Table 25.1 Citizens’ Performances and State Enactments in Indian Village Assemblies 


LOW CAPACITY 


MEDIUM CAPACITY 


HIGH CAPACITY 


CHITHOOR (AP) 
State: Complaint collector 
Citizens: Passive petitioners 


MEDAK (AP) 
State: Complaint collector 
Citizens: Passive petitioners 


DHARMAPURI (TN) 
State: Social reformer 
Citizens: Civic deliberators 


BIDAR (KA) 

State: Scrutinizer 

Citizens: Elite stewards & 
Rude citizens 
COIMBATORE (TN) 

State: Social reformer 
Citizens: Militant deliberators 
DAKSHIN KANADA (KA) 
State: Informant 

Citizens: Pragmatic 


PALLAKAD (KL) 

State: Planner 

Citizens: Benefit invigilators 
KASARGOD (KL) 

State: Planner 

Citizens: Benefit invigilators 


deliberators 


Note: AP: erstwhile Andhra Pradesh; KA: Karnataka; KL: Kerala; TN: Tamil Nadu. 
Source: Sanyal and Rao 2019. 


To understand the influence of literacy (ability to read and write) on civic delib- 
erations, we examined village assemblies differing in literacy levels within the same 
district. The analytic focus was on three attributes: the ways in which citizens artic- 
ulated their demands; their efforts to seek accountability from public officials and 
elected local government leaders with attention to the style of speech in which crit- 
icism was expressed; and the specificity and information, particularly numerical 
information, contained in the framings and justifications. 

Through this process we were able to discover the influence of literacy on citizens’ 
political literacy (knowledge of government operations) and capacity to deliberate. 
Table 25.2 summarizes our findings. We found the state's role to be far more impactful 
than village literacy levels in elevating or suppressing the quality of deliberations. 

Using excerpts from our transcript analysis, we illustrate the talk-based method 
and explain how we used it to characterize the differences in citizens’ performances 
and state enactments in Indian village assemblies. From our state-wise comparisons, 
we showcase village assemblies in the districts of Chithoor in erstwhile Andhra 
Pradesh and Dharmapuri in Tamil Nadu. Despite being from geographically contigu- 
ous blocks and having a shared cultural history, there were dramatic differences in 
citizens’ performances and state enactments, revealing a large divergence in the reach 
and effectiveness of deliberative democracy. In Chithoor, Andhra Pradesh, the state 
as a complaint collector and citizens as passive petitioners are typified in the excerpt 
below. Villagers voiced their demands and grievances in brief utterances, with- 
out providing specific information and devoid of factual details. The political head 
(sarpanch) played a largely ceremonial role, inviting citizens to express their concerns 
and dismissing each concern with cursory promises to communicate the problems 
to higher authorities. 
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Table 25.2 Differences in Citizens’ Performances and Gaps in Political Literacy 


DISTRICT (STATE) LOW LITERACY MEDIUM HIGH LITERACY 
LITERACY 
CHITTOOR (AP) Citizens: s 
Citizens: Passive Passive petitioners 
petitioners 
BIDAR (KA) Citizens: Elite Citizens: Strident Nil 
stewards & Rude deliberators 
citizens (more anarchic & 
Wide Gap hostile) 
DHARMAPURI (TN) Citizens: Civic Citizens: Civic a 
deliberators deliberators 
Narrow Gap 
COIMBATORE (TN) Nil Citizens: Militant Citizens: Militant 
deliberators deliberators 
Narrow Gap (more acrimonious 
& hostile) 


* Note: Sample too small, hence eliminated from comparison. 
Source: Sanyal and Rao 2019. 


SARPANCH: Today we are conducting this gram sabha to discuss the problems in our 
village and the various activities we have undertaken so far. You can express your 
problems here. 

VILLAGER (YOUTH COMMUNITY MEMBER): There is no proper community hall in this 
village for holding meetings or events. We should construct a community hall. 

SARPANCH: I will inform the government to construct a community hall and to provide 
all facilities to conduct meetings. I will try my level best to construct a community 
hall. 

VILLAGER (SC): There are no cement roads in the village. Cement roads should be laid 
on all the village streets. 

SARPANCH: Wherever we don’t have the cc (concrete) roads, I will try and get them 
constructed at the earliest. 

VILLAGER (sc): There are electricity poles on the streets, but the lights are not there. 
Should arrange for the lights. 

SARPANCH: I will arrange for streetlights very soon. 

VILLAGER (sc): In the village some people have huts. About fifty families have no houses 
to stay. So you should construct ‘pucca’ (permanent) houses for all the house-less 
people. We have permission to construct houses on the hill but there is no road. 

SARPANCH: I will discuss with the government officials about this problem. 

VILLAGER: We don't have a proper cemetery or graveyard in the village. Sometimes the 
adjacent villagers throw the dead bodies on the outskirts of our village, and this leads 
to health problems for our children. 
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VILLAGER: We have complained to the Panchayat office, but till now there is no solution. 
They are threatening us. 
SARPANCH: I have given a complaint to the collector regarding this but nothing has 
happened, and I am helpless regarding this issue. 
(Kalyanapuram, Narayanavanam) 


As is evident, village assemblies in Chithoor were vacuous governance rituals, 
thoroughly devoid of deliberations. The lack of transparency from the state kept cit- 
izens under a sort of blindfold, leaving them suppliant, submissively rehearsing a 
litany of complaints with little or no effect. The citizens’ inability to raise pertinent 
questions, seek information, or hold their elected leaders accountable for their per- 
formance was painfully apparent. We interpret these features of citizens’ participation 
to reveal a lack of oral competency to engage in deliberative democracy. 

Village assemblies in Dharmapuri, Tamil Nadu, stood in stark contrast to the above 
pattern. Every single assembly in this district started with an announcement of the 
meeting agenda, which included a range of clearly specified topics that were set by 
the state as governance priorities to be discussed at the meeting. The agenda typically 
included village development priorities. There could be as many as ten or twenty 
agenda items. A substantial part of the discussion was devoted to these themes. 
The agenda items were meant to raise public awareness of state-sponsored develop- 
ment schemes and to encourage villagers to adopt them. Panchayat officials reported 
on their implementation and checked villagers’ compliance. Public officials made 
determined efforts to persuade villagers through moral and pragmatic arguments to 
comply with state-sponsored schemes. This exercise exemplified the social reformer 
state that was using these deliberative forums for public persuasion to bring its own 
vision of village development to fruition. 

The citizens, whom we labelled as civic deliberators, consistently displayed the abil- 
ity to engage in sustained discussions on shared problems. Though a substantial part 
of the time was consumed in deliberating on the agenda items, citizens were also 
capable of pushing aside these state-prioritized goals and introducing onto the dis- 
cussion floor demands and concerns that mattered more to them. The excerpt below 
is one example out of many where citizens engaged in a sustained discussion on water 
shortage. The discussion ended with a pair of villagers offering financial help, and the 
elected leader agreeing to move ahead with assistance from them. The cooperative 
discussion led to a creative solution being proposed: 


VILLAGER (SPEAKER 1): There is no water in the village. What is the panchayat plan- 
ning to do? There is no water in the tank. 

VILLAGER (SPEAKER 2): Lake should be deepened. This is important. Plumber is not 
attending to the fault properly. 

VILLAGER (SPEAKER 3): We need an overhead water tank near Thimmarayaswamy 
temple. People and cattle face much difficulty for water. Ministers and MLAs 
(members of legislative assembly) have not taken any steps! 


Paromita Sanyal 365 


VILLAGER (SPEAKER 4): As much as I know, there is problem of water and elec- 
tricity supply. Water problems are more severe. There is water connection from 
Chinnakothur (nearby village). But somebody has stolen the delivery line since past 
five years. Now the President has to spend Rs 2,000-3,000 to replace the steel pipe 
connection. Therefore, we request that a pump room with a bore well should be 
constructed in Bustalapally itself, like the one provided in the public bathroom. 

MALE PRESIDENT: I will inform the BDO (Block Development Officer) for necessary 
action. 

... (Other demands are expressed and responded to) 

VILLAGER (SPEAKER 15): We need public toilet. There is no water in the water tank. 

VILLAGER (SPEAKER 16): There is no water in the lake even, then how can you expect 
water in the tank. 

VILLAGER (SPEAKER 17): We need cement storage tank at the ground level. 

VILLAGER (SPEAKER 18): If we have water in the overhead tank, then where is the need 
of smaller ground-level tanks! 

PRESIDENT: We can build smaller ground-level tanks only with Panchayat funds. But 
the panchayat does not have sufficient funds. 

VILLAGER (SPEAKER 20): Even if we have water tank, there is no good water, and canal 
water is not good. We also need drainage. That is what I request the President and 
vice president to look into. 

PRESIDENT: Already all our efforts to build the drainage system could not be carried out 
because people did not give land, and they themselves directed the drainage water 
along the roads. Even the drain water pipelines laid were stolen. 


VILLAGER (SPEAKER 22): We will be very happy if drinking water facility for us and our 
cattle is provided by way of water tubs for the cattle and water tank for us. 

PRESIDENT: We can do all these things if we get revenue for the panchayat. 

VILLAGER (SPEAKER 23): We are not asking the panchayat to do this. We are asking the 
government to do this. 

PRESIDENT: Okay, we will also approach the government for assistance. 

VILLAGER (SPEAKER 24): Our most important demand is we should get uninterrupted 
water supply. Whether you change the motor or you repair and change the pipeline, 
we don't bother, but we want water. 


VILLAGER (FEMALE) (SPEAKER 29): Is it not your responsibility to build the overhead 
tank? 

PRESIDENT: No, it is the water board’ responsibility, and it is asking for commission. 

VILLAGER (SPEAKER 30): Is it 20% (commission)? 

PRESIDENT: No, it is 10%. It comes to Rs. 20,000. So I came back. 

VILLAGER (SPEAKER 31): Sir, we two are the temple trustees. We will give Rs. 20,000. 
You get the sanction. 

PRESIDENT: Yes, you also come with me. We will give the money and get the sanction. 

(Bustalapally, Shoolagiri) 
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Citizens here were clearly capable of pursuing information by asking pointed ques- 
tions of their elected representatives. They strongly expressed dissatisfaction with 
the non-performance of high-level representatives, like ministers and legislative 
assembly members, who received budgets designated for the development of their 
constituencies. In addition to being able to critique and force public accountability 
from the local government, citizens also demonstrated their capacity to make con- 
structive suggestions for what needed to be done to solve the shortage of water supply 
(deepening of the lake, building ground water storage tank, and constructing a pump 
room with a well). The deliberation over ends and means of village development 
combined moments of cool-headed consideration of alternatives alongside contes- 
tation and disparagement of public officials and leaders. Citizens displayed a high 
level of oral competency by being in command of factual information regarding pub- 
lic infrastructure problems (stolen pipeline, cost of replacement, plumber doing an 
unsatisfactory job), which they effectively marshalled into their arguments. Through 
this comparative analysis and interpretive characterization of citizens’ performances 
and state enactments, we were able to discover the enormous difference the state 
could make as a facilitator or inhibitor of deliberative democracy. 

In a second project (Sanyal et al. 2019), using the original data described above, 
we examined women's participation in village assembly deliberations. Our particular 
interest was to explore whether there were differences in concerns voiced and com- 
municative styles between women who were members of self-help groups (SHGs, 
which are micro savings and lending groups formed under state poverty alleviation 
programmes) and unaffiliated women. For this analysis, the transcripts were coded to 
capture various dimensions of women’s participation. Issues raised by women were 
inductively coded and generated several categories (civil works, water, health and 
sanitation, jobs and entrepreneurship, finance, and civics and participation). Narra- 
tive strategies the women used were also coded by analysing how women presented 
an account of what they wanted, or the problems they were facing, and whether they 
followed up these statements with a demand for remedial action. The sub-codes that 
were developed inductively were the following: command (‘do x’); complaint (‘you 
have neglected to remedy problem x’); demand (‘you have to give me/us x’); descrip- 
tive (‘there is a problem in our area with the quality/delivery of x’); need (‘I need/want 
x, ‘I don't have x’); query (‘when will funds for x be allotted?’); request (‘please give 
me x’). Each of these styles, for us, represented varying degrees of discursive power. 
For example, statements of need and request were considered less powerful narrative 
strategies reflective of petitioner mindsets, while complaints and commands were 
taken to be reflective of a sense of power to hold the state accountable. Quality of 
participation was assessed by coding every event of a woman speaking into high and 
low, based on a score derived from the issues raised and the narrative strategies used 
to communicate the issues. The score itself was generated by the coding software 
NVivo, but fundamentally based on our hands-on coding of the number of issues 
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raised in a speech and the variety of narrative styles used by the speaker. A con- 
sistent interpretation allowed us to form a rough assessment of women’s quality of 
participation. 

From this analysis, we found a stylistic difference between SHG women and unaf- 
filiated women. SHG members made demands for government action (not requests 
or passive expressions of need), preceding the demand with informative descriptions 
of the problem and past inaction, in effect using these statements as reasons justify- 
ing their demands. Importantly, they used a public goods framing in voicing their 
problems and accompanying demands. These features meant that, in our coding sys- 
tem, they were coded as displaying a higher quality of participation at a higher rate 
than non-member women. Unaffiliated women were divided almost equally between 
high and low qualities of participation. We also found that in states with a relatively 
high baseline of women’s verbal participation, the difference associated with SHGs, 
although significant, was less dramatic. Based on this finding we tentatively con- 
cluded that any intervention seeking to promote women’s vocal participation was 
likely to have more dramatic effects where the baseline of women’s participation was 
low, all else remaining the same. 


Conclusion 


The qualitative method outlined here excels in its ability to examine deliberative 
institutions in graphic detail (what is said, by whom, and how it is said) and in its 
interpretive strength to make sense of those utterances from a performative stand- 
point. Other methods may be better suited to accurately measuring the quantum of 
speech by particular groups, by gender, caste, race, or other identity-based criteria. 
For example, some studies have quantitatively analysed citizens’ and women’s speech 
in Tamil Nadu village assemblies (Ban et al. 2012; Palaniswamy et al. 2019). But if the 
goal is to identify communicative styles and narrative strategies used in deliberation 
and to understand them as part of rhetorical moves and performances, then one must 
rely on qualitative methods such as the one proposed in this chapter. Researchers 
have to be willing to make subjective assessments during the coding and interpretive 
process. These subjective assessments and interpretive characterizations must be car- 
ried out with care. They have to be adequately substantiated with discursive evidence, 
and evidence in favour of and against must be given systematic consideration. Yet, no 
matter how rigorous the procedure, to use the talk-based method, researchers must 
be comfortable embracing the subjective nature of the method. 

Like most methods, the talk-based method is not immune from weaknesses. The 
biggest limitation is that the method is solely focused on analysing what is said on 
the floor of civic deliberations. It cannot tell us anything about the outcomes of these 
deliberations for civic life—what action was taken as a result and its impact on the 
quality of life in the community. For that kind of analysis, the talk-based method 
must be paired with longer-term fieldwork that follows discussions in the deliberative 
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space into the community and traces the extent to which they result in action or fizzle 
away. Another limitation is the difficulty of using this method on large-scale data to 
study if people change their opinions and preferences on public matters in the course 
of deliberation. For that kind of analysis to be possible, the transcripts would need 
to accurately identify each speaker every time they spoke in a deliberative setting. 
This would only be possible for small-sample analysis under very particular condi- 
tions. For instance, this would be feasible in smaller settings where there is a rule or 
custom of turn-taking in speech and people do not talk simultaneously, or collec- 
tively agitate on the floor of the discussion. Moreover, such opinion- or preference 
tracing would only be manageable in unidimensional discussions where there is only 
one focal topic that is being discussed and debated. Current limitations offer oppor- 
tunities for future improvements. Deliberation scholars are invited to build upon this 
talk-based method to realize its potential more fully. 
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26 
Media Analysis 


Rousiley C. M. Maia and Tariq Choucair 


Deliberation occurs in a diversified media landscape. Hybrid communication that 
combines broadcast, print, and online communication transverses social wholes, 
posing ever greater challenges for researchers to develop methods for captur- 
ing media-based deliberation. Traditionally, communication scholars engage with 
inquiry into media communication at individual, institutional, and social levels. 
Whereas some approaches are quantitative-oriented, including methods such as sur- 
veys, experiments, and content analyses, other approaches are qualitative-oriented, 
based on visual observation, interviews, discourse analysis, and historical analysis. 
There are, indeed, many ways to study media and deliberation. 

This chapter focuses on content analysis. We define this research method briefly 
as the ‘systematic, objective, quantitative analysis’ of communication content or 
message characteristics (Nuendorf and Kumar 2015, 1; see also Krippendorff 2012; 
Neuendorf 2017; Riffe et al. 2019). This method is used for making replicable and 
valid inferences from written texts or transcribed speeches as well as non-textual 
content, such as images, graphical elements, music, and sounds. The term ‘commu- 
nication content, therefore, refers to a broad range of materials such as newspaper 
articles, speeches, advertisements, social media posts, blog or microblog texts, and 
so forth. The effort to identify units for building code schemes for human coding 
requires full specification of the set of categories. Usually, coders are trained to use 
categories to measure differences in content; and this training should lead to inter- 
coder agreement, that is, stability in the protocol application over time. Then, the 
analysis of relationships involving these categories is often based on statistical meth- 
ods. If the categories and rules are comprehensive, they should allow replicability 
according to the standards of reliability; and the patterns observed are likely to be 
valid in a more general way when transported to different contexts (Neuendorf 2017; 
Krippendorff 2012). Recently, computer-aided textual analysis has enabled the cod- 
ing of massive volumes of material, by accelerating data collection and reliability 
testing. With technological development, the use of content analysis has become 
more practical and opened up new venues for more sophisticated combinations of 
linguistic and visual cues with statistical analysis. 

Our definition of content analysis works towards an integrative approach, which 
means combining it with methods like surveys, experiments, network analysis, and 
frame analysis to examine the roles that media play in deliberation in different 
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political contexts. Following the systemic turn, scholars have become particularly 
concerned with linkages between micro and macro analyses, so as to find different 
ways to embed deliberative practices or discussions within forums in some larger 
social whole (Bachtiger and Parkinson 2019; Dryzek and Hendriks 2012; Hendriks 
et al. 2020; Mansbridge et al. 2012). This theoretical understanding poses several 
methodological challenges in the field of political communication and media stud- 
ies, given the breadth, scope, complexity, and the speed of transformations in the 
contemporary media landscape. 

This chapter aims to take on this challenge. We argue that the integrative approach 
we offer contributes towards building methodological strategies for studying distinct 
types of media in deliberative systems. We develop this chapter by demonstrating 
various levels of analysing media and public deliberation. In the first section, we 
demonstrate how content analysis can be used to examine mass media material, such 
as newspaper outlets, magazines, TV news, and news media articles. We argue that 
investigating the content of mass media provides insights into the mechanisms that 
enable perspectives, discourses, and claims to become available to broad audiences 
and circulate in the public sphere, transcending space and time. Content analysis, in 
this context, can be integrated with surveys, network analysis, and statistical methods 
to understand the dynamics of public debates. In the second section, we exemplify 
the use of content analysis to examine deliberation in diversified media platforms, 
including websites, social network sites, weblogs, and micro-blogs. We argue that the 
growing literature in digital media has provided ever more precise characterizations 
of online communication and distinct conditions of deliberative engagement, not 
only in deliberatively designed forums but also in other sites of democratic systems. 
In the final section, we demonstrate how content analysis can be adaptive across set- 
tings, in both micro and macro levels of analysis. We combine content analysis with 
statistical methods and frame analysis to reveal a multilayered analysis of deliberation 
in different contexts. This can be done by using less standardized coding typologies 
and ramified analytical strategies to track media dynamics. We outline some research 
designs to illustrate the viability of this approach. 


Analysing Mass Media Material to Map Debates 
in the Public Sphere 


Deliberation, conceived as a society-wide practice, requires argumentative exchange 
in public. Assessing broader acts of reason-giving and justification in the public 
sphere is complicated and underdeveloped, but there are several advances in this 
field from which empirical researchers can draw. Traditionally, scholars have inves- 
tigated how a specific type of media, such as newspapers, TV programmes, and radio 
talks, could operate as forums for civic debate, by selecting topics and employing a 
set of operations to build public discussions. Content analysis is one of the widely 
used methods in this field. 
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There are at least two major approaches that use mass media samples for analysing 
public debates. The first approach seeks to map speakers and arguments that are 
included in media-based communication about an issue-specific controversy. Earlier 
studies investigated the set of speakers and claims that gain access to journalis- 
tic coverage and other forms of mass communication (Page 1996; Bennett et al. 
2004). Later studies became more explicitly interested in capturing clashes amongst 
actors and discourses in journalistic material in order to reconstruct public con- 
troversies. Examples include public debates on abortion (Ferree et al. 2002), gun 
control (Callaghan and Schnell 2001; Maia 2009), same-sex marriage (O’Connor 
2017), voting systems (Pilon 2009), education public policy (Saraisky 2015), and 
technological issues (Peters et al. 2008; Schneider 2008). More recent studies (Haus- 
sler 2018; Lycarião and Wozniak 2017; Maia, 2012a, 2018) incorporated conceptual 
tools of deliberation, that when properly conceived cannot be conflated with face-to- 
face interactions, adapted code schemes (Steenbergen et al. 2003; Steiner et al. 2004; 
Steiner 2012), and distinct forms of operationalization. 

The second approach takes a comparative perspective to investigate the perfor- 
mance of newspapers, magazines, and TV news across distinct media systems in 
different countries. Studies here often focus on variations in national news culture 
and political systems, and explore factors such as autonomy of media organizations, 
format of news, professionalism, audiences, and journalist-source relations (van der 
Wurff et al. 2013; Schmidt et al. 2013; Wessler and Rinke 2014). Comparative con- 
tent analysis is useful for evincing complexities in the relations between journalists, 
political/economic elites, and civic actors (Hallin and Mancini 2004, 2012). It helps to 
elucidate broader political and social structures that shape news practices and affect 
different deliberative dimensions of journalistic performances, which can result in 
greater or lesser plurality of voices and degree of responsiveness in the news media. 
Some studies in this field have moved towards multimodal analysis, by incorporat- 
ing concepts of framing, narration, and visual representation in the code scheme 
(Wessler et al. 2016; Wozniak et al. 2014). 

In this section, we explore methodological strategies related to the first approach. 
Assuming that public debates are processes that unfold over time (Habermas 1996; 
Haussler 2018; Mansbridge et al. 2012), our research group (Maia 2009, 2012a; 
Marques and Maia, 2012a; 2012b; Cal and Maia, 2012; Mendonça and Maia, 2012) 
have developed a set of indicators for capturing different dimensions of media 
debates. 

The original code scheme included four indicators: 


1. Inclusivity and characterization of participants. Who gains access to media are- 
nas? What are the inclusion criteria in journalistic coverage, and how much 
space or time is allocated to speakers? This variable captures the inclusion of 
social actors and their claims in specific media. 
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2. Use of arguments. Do participants present reasons in support of their views, 
preferences, recommendations in public? Is media-based communication 
grounded on justification and demonstration? This variable enables the map- 
ping of con and pro arguments put forward in the media arena regarding a 
specific controversy. 

3. Reciprocity and responsiveness. Is there any dialogue or any possibility of mutual 
response in public amongst speakers with different views? Is there reference to 
utterances of other actors? Who responds to whom in media-based commu- 
nication? This variable aims to identify patterns of discursive interactions and 
rebuttals on a large scale. 

4. Reflexivity and reversibility of opinions. Are changes in position or preference 
observable? This variable is related to processes through which participants 
may review their opinions and arguments in public. 


Building on previous studies (Ferree et al. 2002; Peters 2008; Peters et al. 2008; Wessler 
2008), we argue that media debate dynamics can be observed at different levels: from 
the utterance in a given news media or different outlets, in varying time scales (over a 
short-term or a long-term period). To capture inclusivity, ‘who speaks in the media, 
the coding scheme needs to be adapted for the type of media under scrutiny. In news 
media material, for instance, speakers’ utterances can be tracked either through direct 
quotations that preserve the original wording, or through summarized speech that is 
attributed to sources. A single news article can weave together voices of diverse actors, 
and boundaries between the voices of reporters and the persons reported are main- 
tained to varying levels (Maia 2012a; Wessler 2008). Moreover, media professionals 
grant differentiated ‘standing’ to speakers, that is, acknowledgement of the value of 
one’s voice—in a way more or less indexed to prominence, prestige, or reputation 
enjoyed by a certain actor/group in a given society (Bennett et al. 2004; Ferree et al. 
2002). Speakers’ utterances may assume different meanings depending on the loca- 
tion in the media context (Bennett et al. 2004; Esser and Strömbäck 2014; Ferree et al. 
2002). While official sources, such as spokespersons are often prioritized (Entman 
2004; Tresch 2009), the composition of sources can vary widely between media sys- 
tems in different countries, and across issues and circumstances in the same country 
(Dimitrova and Strömbäck 2009). 

Content analysis becomes especially interesting if we categorize the speakers 
according to their institutionally defined roles. Here, Peters’ (2008) and Habermas's 
(2006, 2009) centre-periphery model of circulation of political power provides use- 
ful guidelines to classify speakers in news stories, in accordance to: (a) agents of the 
executive body; (b) legislative houses; (c) judiciary; (d) political parties; (e) experts; 
(f) media agents, such as journalists, editors, commentators; (g) entrepreneurs; 
(h) organized civil society, advocacy groups, and non-governmental associa- 
tions; (i) intellectuals, artists, and celebrities; (j) church; and (k) ordinary people. 
Investigating the composition of speakers in different sections of the newspapers 
(editorials, commentaries, reports, interviews, and news stories) and examining 
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news-making operations (such as procedures to establish opinions’ hierarchy, to 
roll up argumentation, to grant legitimacy to claims, etc.) offers important cues for 
understanding professional communicators’ attempts to shape the overall course of 
a debate (Bennett et al. 2004; Esser and Strömbäck 2014). Citizens’ opinions are typ- 
ically located in less important spaces in news stories (Hopmann and Shehata 2011; 
Ferree et al. 2002). Yet, the participation of civil society actors in many important 
debates occurring in the media arena is not as marginal as is usually thought. A 
number of studies have demonstrated a greater presence of civil society actors in 
challenger debates and the importance of the themes, claims, and criticisms they 
address to political and economic elites (Haussler 2018; Peter et al. 2008; Schneider 
2008). 

Content analysis is also valuable for investigating the types of arguments dis- 
played in public communication, as related to values, interests, vocabularies, and 
performances of different categories of speakers. The Discursive Quality Index 
(Steenbergen et al. 2003; Steiner et al. 2004; Steiner 2012) has been adapted to exam- 
ine the structure of justifications—their logical connection and level of completeness, 
and the content of reasons alluding to the common good or abstract principles. To 
map the discursive space in more detail, one can include a list of pro and con argu- 
ments in the code scheme (Ferree et al. 2002; Maia 2009, 2012). Toulmin’s model of 
argumentation (Toulmin 2003 [1958]) has also provided guidelines to capture types 
of evidence given in arguments (for example, empirical, statistic, research findings, and 
legal evidence) and types of warrants (for example, conditional, analogy, value-based, 
meta-proposal) (Adams 2014; Karpowitz and Raphael 2014; Maia et al. 2020a). 

An effective way of examining the complexities of reciprocity and reflexivity in 
the public sphere is to focus on sequential moments of media-based communica- 
tion. In longitudinal studies, content analysis helps in assessing changes in discourses 
over time. Depending on the research goals, data collection of news material can be 
planned to cover varying periods of time—weeks, years, or even decades. By sequenc- 
ing justifications offered by actor categories in the news media over time, conflictual 
or co-operative relations become apparent (Nerlich and Jaspal 2013; Michailidou 
2015). One can assess whether speakers make explicit reference to other actors’ argu- 
ments, and shift emphasis or re-balance their positions in the face of other actors’ 
claims (Haussler 2018; Maia 2009, 2012a). In the aggregate level, the voices, claims 
or discourses that either grow stronger or decline throughout the analyzed period 
can also be observed (Ferree et al. 2002; Schneider 2008; Kirilenko and Stepchenkova 
2012; Saraisky 2015). 

Understanding correlations between actors’ argumentation in media communi- 
cation and impacts on institutional decision-making helps to illuminate agreements 
and sorts of justification that become more collectively acceptable as discussions 
progress (Baumgartner et al. 2008; Boykoff 2013; Ferree et al. 2002; Haussler 
2018; Maia 2009, 2012a, 2014). As will be discussed later, mass media-based 
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communication can be combined with deliberation acts observed in parliament or 
civic forums, to advance analysis of patterns of interactions amongst actors in various 
settings. 


Deliberation in Diversified Online Platforms 


Content analysis has been widely employed in the field of online deliberation. The 
first step in the process of analyszing deliberation through content analysis is to 
understand the affordances of online communication. One should investigate, for 
example, the level of identifiability/anonymity of discussants, the format of com- 
munication based on synchronous/asynchronous interactions; the presence/absence 
of moderation; the information available for forum users or technical equipment 
provided for communication (smartphone devices, type of Internet connection, web- 
cams, amongst others) (Esau et al. 2017; Kies 2010; Maia 2014; Maia and Rezende 
2016; Strandberg and Grönlund 2014, 2018; Stromer-Galley 2007). By understand- 
ing this, we get a sense of how online discussion is shaped by the design and purpose 
of the forum. For example, one can ask whether a forum fosters a deliberative stance. 

After understanding the affordances of digital settings, the second step is to analyse 
the content of communicative exchange taking place there. Frequently, content anal- 
ysis involves building representative samples of digital material (such as posts and 
forum users’ comments) to measure the characteristics of communication reflecting 
deliberative dimensions. To develop code schemes, scholars usually focus on delib- 
erative dimensions, such as expression of considered opinions, levels of justification 
supporting positions, expression of civility or respect towards others and arguments, 
levels of disagreement and rebuttals, reciprocity, appeals to the common good, use 
of personal stories, revision of views, and opinion change (Dahlberg 2001a, 2001b; 
Esau et al. 2017; Janssen and Kies 2005; Kies 2010; Stromer-Galley 2007). To analyse 
the deliberative quality of online discussions, code schemes have often adapted the 
Discourse Quality Index (Steenbergen et al. 2003; Steiner et al. 2004). Yet, particular 
attention should be given to features of the digital environment and variations in the 
context of interactions. 

This means that the variables of the code scheme should be constructed in such 
a way as to apprehend the specificities of online discussions. When analysing news 
website spaces for comments, for instance, one should identify whether a partici- 
pant is responding to the news story or to other participants’ remarks (Kies 2010; 
Stromer-Galley 2007). In digital forums, it is also important to observe to what extent 
participants are responding to an existing thread or creating a new one (Kies 2010; 
Stromer-Galley 2007; Bravo and Saez 2016). These two aspects help in identifying 
and measuring reciprocity, so as to understand who is interacting with whom, and 
in what ways, and whether the content is reciprocally shared. Discussants in digi- 
tal settings often use hyperlinks to address content displayed in other spaces, such 
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as news portals, videos on distinct platforms, and even to other discussion threads. 
Identifying and analysing the hyperlinks ‘allow us to see connections among sites of 
deliberation being made at a granular level’ (Lyons 2017, 8). Moreover, online forum 
users, in contrast to face-to-face forum participants, use sources by directly ‘liking’ 
messages, building on previous comments or complementing content with their own 
opinions and arguments (Kies 2010; Stromer-Galley 2007; Bravo and Sáez 2016). 
These are crucial elements to investigate because social media ‘likes’ and shares, 
despite not directly explaining deliberative engagement, enable us to make more fine- 
grained interpretations of the scope and scale of a given online discussion, within the 
ecosystem of referentiality and reflexivity in digital environments. 

Since the growing literature in digital media has become ever more complex and 
diversified, it defies an easy summary of the distinct ways in which content analysis 
has been used in this field. Depending on the research goal, scholars have employed 
content analysis to investigate: 


(1) the effects of specific designs and affordances (such as identifiabil- 
ity/anonymity and heterogeneity/homophily) on political discussions 
(Lewinski and Mohammed 2012; Esau et al. 2017; Halpern and Gibbs 2013; 
Mitozo and Marques 2019); 

(2) distinct dynamics of civil/uncivil discussions and disrespect in spaces har- 
bouring like-minded people or adversarial discussion on YouTube, blogs, and 
Facebook pages (Molaei 2014; Muddiman et al. 2017; Maia and Rezende 2016; 
Rossini and Maia forthcoming); 

(3) factors that enable forum users to engage in a more egalitarian conversation 
on news websites and social media networks (Maia 2014; Rowe 2015); 

(4) types of outcomes of online discussion, concerning their influences on polit- 
ical policies and opinion changes (Strandberg and Grönlund 2012; Filatova 
et al. 2019; Bravo and Sáez 2016); 

(5) group polarization, disagreement and toxic discussions favouring intolerance 
and distrust of political institutions (Smith 2019; Maia et al. forthcoming). 


It is worth noting that scholars investigating online deliberation have recently moved 
towards comparative content analysis, including more components and connections 
between data sets, which is beneficial for producing multifaceted descriptions and 
more nuanced evaluations of each component. While earlier studies have analysed 
only the verbal content of online discussions, later works have become more atten- 
tive to contexts and goals of different online settings to better understand the wider 
implications of their findings. For instance, Esau et al’s (2020) study has differenti- 
ated between highly formal (a government-run consultation platform), semi-formal 
(mass media), and informal (social media) arenas, and employed content analysis 
to examine an issue-specific debate (German immigration policy) to track varia- 
tions regarding rationality, reciprocity, respect, constructiveness, storytelling, and 
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expression of emotion in these settings. Maia et al. (2020b) have systematically anal- 
ysed the relationships between disagreement and reason-giving in online discussions 
about a controversial issue (lowering the age of criminal responsibility) in arenas that 
serve distinct functions in the political system (legislative public hearings, news web- 
sites, and an activist social media page [Facebook]). The authors have also traced the 
implications of these practices for public reasoning. A growing body of studies has 
produced cumulative knowledge on practices of political discussion and deliberation 
underlying political participation and information seeking, and their associated out- 
comes. To date, studies that employ content analysis to map the systematic linkages 
between online discussions and broader components in the deliberative system are 
still rare. 


The Integration of Content Analysis with Other 
Methods 


So far, we have highlighted that studies on political communication and delibera- 
tion have become more methodologically sophisticated through a mixed methods 
approach. In this section, we argue that media research on deliberation can be 
fruitfully expanded by combining content analysis with additional data derived 
from surveys, face-to-face discussions in legislative or civic forums, frame anal- 
ysis, statistical methods, and so forth. The integrative approach offers a tool for 
identifying relationships between political messages in media (news stories, social 
media, and Twitter, for instance) and other communicative practices from political 
representatives, experts, civil society organizations, and ordinary citizens. Research 
incorporating a mixed methods approach enables a better investigation of multiple 
venues for public information and discussion, traversing mass media and online set- 
tings. To be sure, researchers are expected to develop theoretically informed research 
designs and analysis protocols, including a clear description of the rationale for craft- 
ing the research questions or hypotheses, producing the coding scheme, and defining 
the sample and unit of analysis. 

To illustrate this analytical effort, we report studies developed by participants 
of the Research Group on Media and the Public Sphere, from the Federal Uni- 
versity of Minas Gerais (EME/UFMG). Our aim in this section is to illustrate the 
methodological options, trade-offs, and difficulties that arise in actual research. 


Comparing Actors’ Reasoning across Different Arenas 


One possible research design for examining actors’ reasoning within the deliberative 
system is to focus on different well-defined contexts of communication to investigate 
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an analytically equivalent phenomenon under different conditions. A study was 
designed to analyse reason-giving by experts when these actors participate in legisla- 
tive public hearings and when they serve as sources for news media stories (Maia et al. 
2016). A first methodological difficulty arose regarding how to achieve conceptual 
validity, since it can be notoriously difficult to compare reason-giving in legislative 
forums and reason-giving in comments for news making. 

Our first step was to define an issue-specific debate—a controversial bill of law 
proposing to transfer the city bus station from its downtown central location to a 
new district—in order to produce ‘contextualized comparisons’ (Locke and Thelen 
1998, 11). This procedure requires the researcher to make self-conscious and detailed 
considerations about contextual factors and the similarities/differences within the 
phenomenon in the selected settings. In our case, reason-giving is expressed in dif- 
ferent forms in legislative public hearings and news media stories, and these forms 
needed to be conceptually and empirically distinguished. Our sample involved tran- 
scripts of three public hearings over two years and news stories concerning this issue, 
collected from three local newspapers during the same period. To produce sample 
equivalence we defined claims, that is, speech acts containing a demand (Steiner et al. 
2004), as the unit of analysis. We followed the conventional procedure described in 
the literature on deliberation to identify claims in parliaments, that is, speech acts 
that contained a demand (Steiner et al. 2004, 55). In the news media material, we 
compiled every direct quotation or close paraphrase attributed to a speaker in the 
news text; and the analysis paralleled those of the public hearing transcripts. To reach 
higher levels of measurement equivalence, we compared the experts’ arguments with 
arguments expressed by all participants in the public hearings and news stories alike. 
After clustering similar arguments, we found forty-eight arguments in all material. 

Altogether, 374 claims were analysed, sixty-seven from the public hearings and 
307 from printed newspapers. Following an adapted version of the Discourse Qual- 
ity Index, we elaborated a twenty-two-variable codebook for content analysis. To 
substantiate our study, the content of pro and con arguments was also investigated. 
Our findings revealed that experts played a similar role in both formal micro are- 
nas (public hearing) and informal macro arenas (news media); they predominantly 
used arguments without any explicit reference to the common good and avoided 
engaging with conflicting arguments from third parties in these two settings. Since 
experts used mainly qualified and sophisticated levels of justification in both envi- 
ronments, this study, counter-intuitively, provided evidence that experts did not 
necessarily issue ‘lower-level’ justifications in media-based communication, in front 
of large audiences. Experts typically did not incorporate counterarguments in their 
utterances, and they were not politically accountable when confronted with other 
interests and goals for society, either in the legislative forum or in the journalistic 
arena. Interestingly, experts’ views, however, have been taken up by other discus- 
sants (non-experts), and the majority of speakers disputed their political preferences, 
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mostly using technical-knowledge justifications in both formal and informal contexts 
under scrutiny. 

Another study was designed to map how activists—‘Movimento Tarifa Zero de 
Belo Horizonte’ (Free Bus Fare Movement)—built arguments in three different are- 
nas (Arantes 2017; Maia et al. 2017). The settings were: face-to-face open assemblies 
in public spaces involving the local population; a Facebook page administered by the 
activists; and the news media covering the topic at stake, when Tarifa Zero move- 
ment’s speakers served as a source for journalists. We built a five-variable codebook 
(including the mobilization goals, expansion of the movement, sort of divulgation, 
diagnostics/strategies, arguments), which was applied when examining the tran- 
scripts of the assemblies, the content of news articles, and Facebook posts, covering a 
three-month period. Specific categories were built for specific settings. We analysed 
the topics under discussion in each setting, and classified the content of arguments 
according to the following aspects: evidence (factual statement, story); conclusions 
(proposal, problem definition); and warrants (conditional, value statement, analogy, 
and meta-proposal) (Adams 2014). In addition, we analysed the content of argu- 
ments. This analytical framework revealed the activists’ reasoning variations across 
settings as part of a broader discursive dynamic and their sequential interactions with 
distinct actors in society. 

The type of research reported above does not aspire to select cases that are repre- 
sentative of diverse populations. Process tracking here focuses on processes within a 
particular case, not on correlation of data across cases. Our aim was to uncover, as 
much as test, reason-giving mechanisms that might explain our correlational find- 
ings in distinct settings. This methodological framework can offer rich explanations 
as to the linkages between arenas and a good picture of the complexity of actors’ 
reasoning in distinct settings within the deliberative system. These studies, never- 
theless, reflect a relative difficulty in making generalizations that apply to broad 
populations. 


Comparing Frames and Reasons in Multiple Sites 


Another good example of research design seeking complementarity of methods 
includes articulation of content analysis and frame analysis. In studies about delib- 
eration, this combination is beneficial because it enhances ways of looking at a given 
problem and provides different paths of empirical investigation so as to develop 
more effective explanations. For instance, more conventional content analysis of 
claim justification reveals nuances of the logical structure of specific arguments, ref- 
erences to own-group or other groups, the common good, and abstract principles 
(Steenbergen et al. 2003; Steiner et al. 2004; Steiner 2012). This method alone, never- 
theless, has difficulties in dealing more satisfactorily with issues of macro analysis. 
By contrast, frame analysis enables the analyst to grasp the broader perspectives 
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that underpin: (a) disputed definitions of the problem at stake, (b) attribution of 
responsibility and (c) solutions offered for conflict resolution (Chong and Druckman, 
2007a, 2007b, 2011). Yet, this method fails to show concretely how objections are built 
and how preferences are explained and transformed. By definition, frame analysis 
allows researchers to capture perspectives and interpretive schemes encompassing a 
full range of ideas, as an ‘organizing principle’ that people use to structure the world, 
current events, and their experiences (Druckman 2004; Entman 1993; Goffman 1974; 
Kahneman and Tversky 1984; Maia and Vimieiro 2013; Matthes and Kohring 2008). 
Methodologically, arguments and frames can be comparable in a congruent man- 
ner within settings and across different contexts, allowing for the investigation of the 
phenomena in more detail and accommodating larger interpretations. 

In this line, the study developed by Choucair (2018; see also Choucair et al. 2018) 
examined both arguments and frames employed in discussions in formal as well as 
informal settings. Focusing on an issue-specific debate—a Bill of Law proposing more 
rigid criminalization of abortion in Brazil (Bill of Law 5069/13)—this research anal- 
ysed: (i) speech acts from political representatives in three deliberative meetings of 
the Commission of Constitution and Justice and Citizenship (CCJC) of the Chamber 
of Deputies, where the constitutionality of the Bill was voted; and (ii) posts and com- 
ments regarding the aforementioned Bill on Facebook from diverse actor categories. 
This study used content analysis to investigate whether pro and con justifications, 
expressed in citizens’ informal online discussions, correlate with those vocalized by 
representatives in legislatures at the centre of the political system. A frame analysis 
was also performed in the same material. 

Since this research sought to say something about political representation, concep- 
tualized in terms of discourse (Bohman 2012; Saward 2009; Maia 2012b), analysis of 
discussion within legislatures would not be enough. We sought to create a represen- 
tative sample to allow inferences from a larger population. Data collection included 
Facebook posts over a one-year period, encompassing a wide range of actors, such 
as ordinary citizens, members of social movement organizations, religious leaders, 
experts and political representatives, and celebrities. In total, we mapped 1158 
Facebook pages from these different actors. A total number of 310,151 posts were 
collected from these pages. The estimated number of comments on these posts was 
3,009,677." 

Given the huge number of online settings, the problem of site selection in the dig- 
ital media landscape constitutes one of the central challenges in current research. To 
deal with a realistic number of observations, we built a probabilistic sample of com- 
ments and posts.” To produce conceptual refinement with a higher level of validity 


1 Given the difficulty in collecting such a large amount of Facebook data, we decided to collect only 
posts. Then, comments were collected from the probabilistic sample of posts, which indicated that there 
were approximately 3,009,677 comments in total, with 95 per cent reliability and 5 per cent sample error. 

? We constructed a probabilistic sample of posts (N = 382) and comments (N = 655) with 95 per cent 
reliability and 5 per cent sample error. 
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over a small number of cases, we built a 19-variable codebook to capture the position, 
the arguments (both pro and con) and the elements of the frame—the definition of 
the problem (Entman 1993) divided between the definition of the problem in relation 
to whom, that is, the actor, and the definition of the problem in relation to what, that 
is, the topic (Matthes and Kohring 2008); moral judgements (Entman 1993) divided 
between positive and negative judgements; diagnosis of causes (Entman 1993); and 
suggestion of solutions (Entman 1993). 

By employing both content analysis and frame analysis, this study allowed 
advancement on different fronts. Analysis of arguments indicated that a large num- 
ber of aspects about abortion criminalization were discussed on Facebook, whereas 
political representatives were more strictly focused on the Project of Law 5069. More- 
over, a wide range of arguments was employed by Facebook users, whereas only a few 
of these arguments were expressed by political representatives. These results do not 
necessarily constitute a legitimacy problem, deliberatively speaking, insofar as polit- 
ical representatives may ‘filter’ a multitude of detailed issues regarding controversial 
debates and provide a synthesized argumentation. 

Application of frame analysis, however, revealed important new features of the 
discursive process itself. By examining the frames employed in formal and infor- 
mal settings a notable result emerged: while the abortion issue was mostly framed 
as a problem pertaining to women’s concerns (38 per cent) in Facebook discus- 
sions, legislative discussants rarely used this frame (15 per cent). This discrepancy 
is particularly significant because this was the prevalent framing pattern in citizens’ 
discussions on both sides of the debate, that is, amongst those speakers in favour or, 
as well as those against, abortion criminalization. Thus, this finding suggests a seri- 
ous disconnect in the interpretive scheme employed by citizens and their political 
representatives regarding the causal diagnosis as well as suggested solutions to the 
problem at stake. 


Conclusion 


This chapter explored how different types of media can be analysed in conjunction 
with deliberative theory. We aimed to provide brief descriptions of different types 
of research to highlight the applications for, and main trends in, the use of content 
analysis in a variety of contexts. After surveying studies on mass communication 
and digital media, this chapter indicated some ways to link micro and macro analy- 
ses. We argued that content analysis, when integrated with other methods, helps to 
produce a better understanding of different connections between actors, delibera- 
tive acts/practices, and media dynamics. Researchers should be encouraged to add 
different data to their content analysis whenever possible to take greater account 
of the hybrid and multi-platform media environment. This helps to avoid poten- 
tially limited interpretations occurring at one site only. In general, images and visual 
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elements remain underexplored in studies on deliberation. In future studies, schol- 
ars may focus on variations of content analysis to capture the broader implications 
of visual cues and non-textual ways of producing meaning in public deliberation. 
This requires that scholars advance and refine their concepts and become conver- 
sant with different methods. The integrative content analysis model, for example, by 
combining argumentation and frame investigations, also helps to clarify the strengths 
and limitations of specific analytical tools in order to make more informed readings 
of findings. Obviously, this is a difficult task because content analysis, frame analysis, 
and statistical and formal modelling are becoming increasingly sophisticated and are 
constantly updated. However, this integrative approach also opens up some impor- 
tant opportunities for more collaborative research amongst scholars working with 
different methods. 
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27 
Mixed Methods 


Oliver Escobar 


The areas of inquiry in deliberative democracy are multifaceted and often require 
methodological approaches that can grapple with complexity. This chapter * argues 
that mixed methods are particularly well suited to investigating deliberative democ- 
racy, while noting advantages and limitations of this approach. The chapter reviews 
methodological foundations, outlines basic aspects of research design, and illustrates 
how mixed methods can contribute to deliberative scholarship. 

Mixed methods research (MMR) entails combining qualitative and quantitative 
strands in a research programme where findings and inferences are derived through 
the methodological and/or analytical integration of data. You may integrate in the 
early stages of data generation, for example, using the findings from one method 
(e.g. qualitative observation) to develop the foundations for another (e.g. quantitative 
survey). Or you may integrate later, for example while addressing a research question 
by drawing inferences from both quantitative and qualitative findings. Combining 
methods that generate one type of data, for example qualitative interviews and focus 
groups, constitutes a multi-method design, rather than MMR. Conversely, generating 
qualitative and quantitative data without integrating both strands constitutes a quasi- 
mixed design. Some level of integration across data sources and/or analytical strands 
is therefore what defines MMR (Bazeley 2018; Hesse-Biber and Johnson 2016). 

MMkR has proliferated in applied social science fields such as health, education, 
social policy, and international development, motivated by the need to conduct 
research that can inform policy and practice (Brannen and Moss 2012; Bazeley 2018). 
MMR starts from the premise that social phenomena and lived experiences are mul- 
tidimensional and therefore research can be both limited and limiting if we grapple 
with complexity through a single dimension. MMR invites researchers to ‘the large 
table of empirical inquiry where they may engage with ‘multiple ways of seeing and 
hearing, multiple ways of making sense of the social world, and multiple standpoints 
on what is important and to be valued’ (Greene 2007, 20). 


* I dedicate this chapter to Andy Thompson—friend and mentor in learning, doing, and teaching mixed 
methods. I want to also acknowledge the funding and support from the Edinburgh Futures Institute, Cli- 
mateXChange, and the What Works Scotland programme (ESRC Grant ES/M003922/1). Finally, I would 
like to thank Hans Asenbaum, Nicole Curato, Selen Ercan, and Ricardo Mendonga for very helpful reviews 
of the draft. 
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MMR is underpinned by paradigm pluralism, positing that various worldviews 
may serve as philosophical foundations for research (Teddlie and Tashakkori 2012, 
779). The starting point for a mixed methods study is not a particular scientific 
paradigm (e.g. positivism, constructivism), or their disciplinary variants (e.g. post- 
positivism, interpretivism, critical realism). Instead, MMR starts with the problem 
or question that animates the research and then mobilizes relevant methods. There is 
some disagreement amongst methodologists regarding the ‘incompatibility thesis, 
which argues that mixing is inappropriate due to clashes between the epistemo- 
logical paradigms that underpin qualitative and quantitative methods (see Bryman 
2006). MMR scholars respond to this ‘purist’ stance through frameworks such as 
the ‘pragmatist, ‘dialectical, or ‘transformative’ approaches, which reject the exis- 
tence of inherent linkages between methods and paradigms and provide alternative 
philosophical foundations (Hesse-Biber 2016; Biesta 2010). 

Accordingly, MMR is guided by the research questions and thus unencumbered by 
fixed philosophical or disciplinary loyalties. It transcends the qualitative/quantitative 
divide that fuelled the ‘science wars’ in favour of a pluralistic approach to social and 
political inquiry (Escobar and Thompson 2019, 503-505). MMR therefore accom- 
modates multiple philosophical traditions, theoretical lenses, lived experiences, 
normative perspectives, and methodological approaches to grapple with complex- 
ity and generate a better understanding of the phenomena under investigation 
(Greene 2007, vii). 


Mixed Methods Research and Deliberative Democracy 


These foundations make MMR well suited to the study of deliberative democracy, 
given their ontological, epistemological, and normative coherence. Ontologically, 
the objects of inquiry in deliberative scholarship are multiple, multilevel, multi- 
faceted, and changing. The variety of beings that populate deliberative studies implies 
a diverse ontology that may encompass individuals, groups, artefacts, processes, 
systems, cultures, and so on. Epistemologically, deliberative studies range from pos- 
itivism to constructivism and their variants, thus accommodating various ways 
of knowing (cf. Ercan et al. 2017). Normatively, deliberative scholarship carries a 
commitment to pluralism in values and viewpoints, which is also central to MMR. 
This coherence between MMR and deliberative democracy is unsurprising given 
their shared intellectual heritage from classic pragmatism (Escobar 2017b; Biesta 
2010). Their synergies offer untapped potential as the ‘third generation’ of delib- 
erative scholarship develops an empirical agenda in pursuit of breadth and depth 
(Elstub 2010). Deliberative theory has morphed into a field of applied scholarship, 
where ideas and practices intertwine in growing research, civic, and policy networks 
and across communities of place, practice, identity, and interest. Democratic innova- 
tions are proliferating globally, multiplying experimentation and institutionalization 
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of deliberative processes across policy arenas and levels of governance (Bachtiger 
et al. 2018; Elstub and Escobar 2019; Chwalisz and Cesnulaityte 2020). This is fertile 
ground for the contribution of MMR, given its focus on generating applied research 
and actionable learning. 

MMR fits well with recent calls for deliberative scholarship to be guided by the 
objects of inquiry rather than the habits of disciplinary or methodological silos 
(Bachtiger 2018). Methodological choices have profound consequences in terms of 
the empirical realities thrown into relief and the issues that become matters of pub- 
lic and research concern. For instance, if we choose to investigate solely through a 
quantitative lens, we may lack depth, whereas if we choose a qualitative lens, we may 
lack breadth. Reducing the scope for discovery to the single track of mono-method 
research, or to the confines of a multi-method approach, may limit our capacity to 
investigate complex phenomena. 

Some research questions asked about deliberative democracy require attention to 
both patterns and cases, statistics and narratives, measures and meanings, numbers 
and words. For example, while studying mini-publics we may want to investigate the 
quality and effects of deliberation amongst participants, as much as their personal 
experiences of the process or the perspectives of citizens in broader publics affected 
by the decisions. When researching community deliberation processes, we may want 
to evaluate local outcomes as well as broader impact on policies, political culture, 
and institutional development. Or we may seek to understand the everyday work 
of deliberative practitioners as well as the effects of facilitation practices across com- 
parative cases. The objects of inquiry in deliberative democracy are thus multifaceted 
and often require a varied methodological toolbox. 

MMR can address exploratory, explanatory, and confirmatory questions simulta- 
neously, which allows the generation and verification of theory in the same study 
(Teddlie and Tashakkori 2009, 578-587). Exploratory questions aim to generate new 
insights and are usually open and tentative, seeking to illuminate an issue for which 
there is limited knowledge—for example, how do organizers, experts, and/or partic- 
ipants decide what types of evidence should be presented in a mini-public? In turn, 
explanatory and confirmatory questions seek to explain or confirm insights for which 
there is already a body of evidence—for example, why do some participants change 
their views through deliberation? An MMR project may feature separate strands with 
different questions, as well as overall questions that seek to integrate those strands— 
for example, how do different types of evidence presented in a deliberative process 
affect the participants’ views? 

The added value of MMR is articulated by Creswell and Plano Clark (2007, 
9-10) as follows. First, MMR provides strengths that can offset the weaknesses of both 
qualitative and quantitative research by addressing questions that may not be answer- 
able by either approach alone. Second, MMR accommodates multiple viewpoints and 
encourages dialogue to overcome the (sometimes) adversarial relationship between 
qualitative and quantitative researchers. Finally, MMR is practical and resource- 
ful because the methodological toolbox is wide-ranging. This versatility enables the 
investigation of normative or empirical puzzles that emerge from the deliberative 
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phenomena at hand, rather than limiting ourselves to questions that can be asked 
through our preferred methods. As in the proverb, if you are always holding a ham- 
mer, everything begins to look like a nail. The challenges of investigating deliberative 
democracy require space for creativity and discovery, and this invites us to open up 
the toolbox and look beyond the hammer. 

Nevertheless, there are critiques and notes of caution to temper optimism and 
expectations around MMR (see Hesse-Biber 2016; Ahmed and Sil 2012). MMR is not 
always viable or advisable, nor is it necessarily superior to mono-method or multi- 
method research. Bryman (2008, 624) offers a useful catalogue of challenges. First, 
MMR requires skills for both qualitative and quantitative approaches. Researchers 
must be conversant with basic foundations and research designs for both, and open 
to ongoing development of technical expertise. Second, MMR is usually resource- 
intensive as multifaceted lines of inquiry increase fieldwork strands. Finally, to merit 
effort and investment, MMR must generate findings that genuinely result from inte- 
gration and that could not be gleaned via another approach. These challenges make 
MMR sometimes more viable for medium- and long-term research rather than for 
shorter studies (for practical solutions, see Creswell et al. 2008). 

The study of deliberative democracy offers opportunities for MMR to address 
questions that are unanswerable through a single qualitative or quantitative 
approach. The following suggestions illustrate range and scope. For example, to 
understand the frequency and depth of public deliberation in everyday conversa- 
tions, we may start by conducting a population survey and use the findings to frame 
deeper inquiry through focus groups. Or we may want to investigate the factors 
contributing to variable levels of participants’ influence over the outcomes of delib- 
erative processes. We could use process tracing in a small set of archetypal cases to 
explore key factors and then test the explanatory power of these emerging variables 
by building a large dataset for quantitative comparative analysis. 

Or let’s say that we aim to understand how facilitators address power inequalities 
in group deliberation. We could start with non-participant observation of facilitation 
work in diverse contexts. Over time we could map strategies and tactics and even- 
tually test their efficacy through an experimental design. Finally, let’s imagine that 
we seek to investigate the contribution of public officials to improving transmission 
or translation processes in deliberative systems. We may start by interviewing prac- 
titioners operating at nodal points between civic and official spaces (e.g. engagement 
officers, digital managers, partnership coordinators) and develop an observation 
protocol for ethnographic shadowing of a sub-sample. The resulting dataset then pro- 
vides the foundations for a survey of practitioners across various policy arenas and 
levels of governance, helping to illuminate the role of agency and institutional culture 
in the functioning of deliberative systems. 

These suggestions illustrate the potential of MMR to investigate deliberative actors, 
groups, processes, systems, and cultures. I will return to specific examples after intro- 
ducing how researchers can design their study and undertake analysis drawing on 
mixed methods. 
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Using Mixed Methods: Design, Analysis, and Quality 


The normative and practical considerations that apply to any single method pre- 
sented in this book provide relevant guidance for the individual strands of an MMR 
study. What makes MMR distinct is the combination and integration of qualitative 
and quantitative findings as part of a coherent research design. This section focuses 
on features that are unique to MMR, namely, its signature research designs, analytical 
approaches, and quality standards. 

The first consideration is whether the research question, problem, or puzzle invites 
a mono-method, multi-method or MMR design. A robust mixed methods project 
begins with a clear rationale that explains why MMR is best placed to address 
the question(s). Qualitatively oriented questions often explore ‘specific dynamics 
or processes of everyday life; typically with a focus on hidden or unquantifiable 
dimensions (Hesse-Biber 2010, 43). Quantitatively oriented questions usually seek 
to test relationships between variables, for example checking how independent vari- 
ables (assumed to be determining factors) relate to dependent variables (assumed 
to be effects) (Ibid.). In turn, MMR questions blend empirical interest in mean- 
ings, actions, practices, or interactions as well as causality, frequencies, patterns, or 
correlations (Hesse-Biber 2010, 44). 

A long-standing typology of rationales for MMR outlines five distinct purposes, 
and mixed methods projects are usually underpinned by at least one of these (Greene 
et al. 1989, 259). The first is triangulation, which checks corroboration or conver- 
gence between findings from different methods. For example, when conducting 
research on a mini-public we may check whether findings from observation field- 
notes regarding power dynamics amongst participants are corroborated or disputed 
by the findings from anonymous questionnaires (e.g. Roberts and Escobar 2015). 
The second distinctive purpose of MMR is complementarity, which seeks elaboration, 
illustration, or clarification of the findings from one method with the findings from 
another. For example, one could complement a population survey on deliberative 
attitudes with focus groups or interviews to generate a richer dataset on meanings, 
experiences, values, and narratives (cf. Neblo et al. 2010). The third purpose of MMR 
is development, where findings from one method help to develop another method. 
For example, the findings from studying income distribution and community partici- 
pation in a local area may be used to inform observations and interviews with people 
involved in local deliberation at participatory budgeting assemblies (e.g. Baiocchi 
2005). 

The fourth purpose is initiation, by which the findings from one method inspire 
the use of another method to address a puzzle or contradiction or to elicit a new 
perspective. For example, one could imagine using the most puzzling results from 
a quantitative experiment on public attitudes to evidence to inform qualitative 
action research in an actual deliberative policymaking process. The final purpose is 
expansion, which seeks to amplify the scope of the inquiry by adding new method- 
ological strands. For instance, an ethnographic study of deliberative practitioners 
may be expanded through a quantitative survey of its broader community of practice 
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(e.g. Escobar 2014; Escobar et al. 2018). These different purposes can be articulated 
and combined in various ways depending on the choice of research design—Box 27.1 
offers an overview. 


Box 27.1 Types of Mixed Methods Research Designs 


Adapting and blending the typologies by Creswell and Plano Clark (2011, 69-104) 
and Teddlie and Tashakkori (2009, 2557-2563), we can distinguish five basic MMR 
designs: 


1. In parallel designs mixing occurs more or less simultaneously to answer 
related aspects of the same questions by drawing on both strands (quan + 
qual).” 

2. In sequential designs, mixing takes place in stages, with qualitative and quan- 
titative strands sequenced purposefully so that each informs the next, guided 
by questions that may evolve. Within this type, there are 

a. explanatory sequential designs, with the quantitative strand shaping the 
qualitative (QUAN — qual); 

b. and exploratory sequential designs, where the qualitative strand sets the 
foundation (QUAL — quan). 

3. In conversion designs, mixing occurs when one type of data is transformed and 
analysed both qualitatively and quantitatively (quan—<—qual), for example, 
text may be analysed thematically and then coded numerically to check fre- 
quencies or patterns. 

4. In embedded designs, one strand takes place within the other, for example, an 
interview within an experiment: QUAN [qual]; ora questionnaire within a focus 
group: QUAL [quan]. 

5. Finally, multi-level designs entail larger programmes of inquiry where various 
sequential, parallel, conversion, and embedded designs may be combined. 


Interpreting findings from different strands through combined analysis can be a chal- 
lenging aspect of MMR (for guidance, see Bazeley 2018). While quantitative and 
qualitative data must be generated and interpreted ‘according to their own merits, 
the benefit of mixing methods lies ‘in the way the data are integrated or can be used 
to interrogate each other’ (Brannen and Moss 2012, 799). As noted earlier, integration 


> Developing a terminology for MMR has been an ongoing endeavor in the field (Creswell and Plano 
Clark 2011, 189). This paragraph illustrates a bespoke notation system, where symbols indicate type of 
relationship and capitalization indicates priority. 
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entails mixing strands either during data generation (e.g. findings from one method 
inform the work of another method) or at the stage of systematic analysis (e.g. draw- 
ing inferences from both strands and making sense of their resonance or divergence). 
This work can now be aided by software such as Dedoose, MAXQDA, NVivo, and 
QDA Miner (see Bazeley 2018, 37-49). 

MMR is a craft—that is, flexible, iterative, responsive. Drawing on Greene (2007, 
144-145), Onwuegbuzie and Teddlie (2003) and Bazeley (2018), there are various 
stages and options available for data processing, interpretation, and analysis: 


e Dataset creation: All data are processed according to quality standards appro- 
priate to their respective sources, checking for methodological rigor, and 
organizing the dataset to enable accessibility and reflect range and depth. 

e Data reduction: Data may be reduced through an initial analytical round 

that seeks to render it manageable by generating descriptive codes, frequen- 

cies, descriptive statistics, factors, case summaries, memos, or other ways of 
summarizing or synthesizing. 

Data display: A useful heuristic for analytical integration of data from different 

sources is to develop visual representations such as tables, charts, diagrams, or 

logic models. 

e Data transformation: Quantitative data may be standardized, scaled, factor 
analysed, etc., while qualitative data may be developed into case profiles, the- 
matic maps, critical incidents, chronological narratives, analytical codes, etc. 
MMR offers the option of transforming qualitative data into numbers (quanti- 
tizing) or numbers into words (qualitizing).° 

e Data comparison and/or correlation: This is about exploring patterns and rela- 

tionships in the dataset, for example developing clusters of variables, themes, or 

stories that indicate resonance or divergence. If qualitative data is quantitized, 
we may run tests to check for patterns. If quantitative data is qualitized, we may 
conduct new qualitative coding and analysis. 

Data consolidation: Sometimes it may be possible to combine different types of 

data to create new variables, themes, or datasets. 

Analysis of findings to draw inferences and meta-inferences: This is the process 

by which we arrive at a set of ‘negotiated and warranted’ conclusions (Bazeley 

2018, 277-280), going from findings derived through each method, to infer- 

ences drawn from those findings, and then to meta-inferences developed from 

combining methods. 


In MMR, inferences are conclusions derived from analysing findings from each 
qualitative or quantitative strand, whereas meta-inferences are conclusions gener- 
ated by analysing inferences across strands (Teddlie and Tashakkori 2009, 4900 


° For more information on qualitizing and quantitizing, see Teddlie and Tashakkori (2009) and Bazeley 
(2018). 
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and 2567). Meta-inferences thus epitomize the added value of MMR, that is, claims 
and arguments warranted by interpretive integration. 

The MMR field has developed bespoke criteria to assess research quality. A robust 
MMR study includes: an explicit rationale for MMR; a design outline articulating the 
purpose, sequencing, and priority level for different strands; an overview of data gen- 
eration and analysis for each strand; an indication of where and how integration was 
conducted; reflections on limitations; and an account of meta-inferences or insights 
drawn from mixing methods (O’Cathain 2010). 

Tashakkori and Teddlie (2008) have proposed an ‘integrative framework to assess 
MMR. Data quality is first evaluated according to customary standards—that is, 
if quantitative data are valid/reliable and qualitative data are credible/dependable, 
then the study has ‘high overall data quality’ (Teddlie and Tashakkori 2009, 3493). 
In addition, the integrative framework proposes a new set of criteria specific 
to MMR: 


e Design quality refers to the quality of inputs at all stages, including research 
design, data generation, and analytical procedures (Teddlie and Tashakkori 
2009, 4848). 

e Interpretive rigor refers to the quality of the process of making meaning by draw- 
ing inferences through the systematic linking and interpretation of findings 
(Teddlie and Tashakkori 2009, 4849). 

e Inference quality derives from blending the notions of internal validity (quan) 
and trustworthiness (qual) and is the standard for evaluating the quality of 
conclusions drawn from findings (Teddlie and Tashakkori 2009, 467). 

Inference transferability blends the principles of external validity (quan) and 
transferability (qual) to assess to what extent the conclusions may resonate, 
be applicable, or offer insights in other contexts (Teddlie and Tashakkori 
2009, 470). 


Researchers can sometimes be challenged by seemingly contradictory, divergent, or 
dissonant findings from different strands of an MMR study. This may seem prob- 
lematic, but it can also help to enrich the analysis. Discrepancies between inferences 
force us to re-examine the findings or to ‘create a more advanced theoretical expla- 
nation (Teddlie and Tashakkori 2009, 5191; for practical solutions to this challenge 
see Tashakkori and Teddlie 2008, 116). Divergence and dissonance illustrate the 
capacity of MMR to grapple with complexity by generating ‘puzzles and paradoxes, 
clashes and conflicts that, when pursued, can engender new perspectives and under- 
standings, insights not previously imagined, knowledge with originality and artistry’ 
(Greene 2007, 24). From this perspective, divergence is not a hindrance to be reluc- 
tantly accepted, but something to be embraced as a potential source of analytical 
creativity and depth. All in all, ‘convergence, consistency, and corroboration are 
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overrated in social inquiry (Greene 2007, 144), and engaging with the messiness 
of socio-political phenomena can be one of the joyful and generative challenges of 
conducting MMR. 


Using MMR in Deliberative Democracy Research 


The use of MMR in deliberative scholarship is somewhat sparse. A database search 
of the terms ‘mixed methods’ and ‘deliberation/deliberative’ elicits a limited number 
of publications.* This does not mean that researchers of deliberative democracy are 
not mixing methods, but it does suggest that it is uncommon to articulate those stud- 
ies as MMR. Does this matter? Mixing methods is more than conducting qualitative 
and quantitative research in one study. There are philosophical, technical, and ana- 
lytical considerations to be heeded, and it is hard to see how this can be done without 
reference to basic tenets of MMR. 

There are exceptions to this limited use of explicit MMR approaches to study delib- 
erative democracy. For example, there is growing work on ‘moral case deliberation’ 
within the context of health policy and practice (Spijkerboer et al. 2017); studies of 
‘deliberative contestation’ in local development (Gibson and Woolcock 2008) or civil 
society deliberation in post-conflict justice (Kostovicova 2017); and research about 
emotions in deliberative processes like the Citizens’ Initiative Review (Johnson et al. 
2019). These studies use MMR to combine a range of methodological angles in order 
to make sense of complex processes, actors, and contexts. 

Deliberative scholars have also developed approaches that incorporate princi- 
ples and practices of MMR. For instance, the Discourse Quality Index (Steenbergen 
et al. 2003), and its conceptual expansion via Deliberative Transformative Moments 
(Jaramillo and Steiner 2019), illustrate the analytical possibilities opened up by quan- 
titizing qualitative data. Another example is Q methodology, which creatively blends 
qualitative and quantitative work to investigate discourses (e.g. Niemeyer 2019) and 
practices (e.g. Durose et al. 2016). There is also scope for expanding MMR as com- 
parative approaches to the study of democratic innovations proliferate (Ryan 2019; 
Boswell et al. 2019). 

The remainder of this section introduces examples that illustrate three MMR 
designs: parallel, sequential, and multi-level. 


* Search conducted in October 2019 using the Web of Science Core Collection database. The paired 
terms were ‘mixed methods’ and ‘deliberat*’ (using a wild card) and generated 161 entries. Titles and 
abstracts were screened for relevance, reducing the sample to twenty-three publications where MMR was 
explicitly used to investigate aspects of deliberative democracy. Almost half of the entries were in health- 
related fields (47 per cent), with the rest spread sparsely across education, environment, communication, 
justice, computing, international development, urban planning, and methodology journals. This was not 
a full systematic review, but it offers a proxy to illustrate limited usage of MMR in deliberative scholarship, 
particularly within political science and public administration. 
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Example 1. Parallel Design: Three Citizens’ Juries 
on Wind Farm Development 


The first example is a study of three citizens’ juries tasked with developing princi- 
ples to guide wind farm development in Scotland (see full report in Roberts and 
Escobar 2015). The study featured a parallel design repeated in the three sites that 
combined quantitative (i.e. panel survey at four time points) and qualitative data 
sources (i.e. non-participant observation fieldnotes, facilitation debriefs, evaluation 
reports, artefact analysis, presenter interviews). The purposes for mixing methods 
were complementarity and expansion in order to generate evidence about the complex 
intertwining of inclusion, interaction, deliberation, and influence within the process. 
Citizens’ juries, like other mini-publics, provide excellent opportunities for MMR 
because parallel strands can be developed to grapple with phenomena unfolding 
simultaneously within and across sites. 

In this project, the parallel design generated a large dataset that enabled the study 
of multiple dimensions, including: the politics and logistics of organizing mini- 
publics on contested issues involving multi-stakeholder policy networks (chapter 3); 
the evolving internal dynamics of citizen participation at the juries (chapter 4); the 
quality of public deliberation throughout the process (chapter 5); the conclusions 
and outputs developed by the mini-publics (chapter 6); the subtleties of delibera- 
tive learning and contestation while engaging with evidence and expertise (chapter 
7); factors in opinion formation, consolidation, and change (chapter 8); participants’ 
experiences of deliberative work and the development of civic skills and attitudes 
(chapter 9); and a set of meta-inferences to inform recommendations about the role 
of mini-publics in decision-making—including the intricate relationship between 
policy context, process design, and public legitimacy (chapter 10). 

The report illustrates the capaciousness of MMR, in this case generating insights to 
address twenty-four research questions ranging from micro dynamics in deliberation 
to macro dimensions in the institutionalization of deliberative processes. 


Example 2. Multi-level Design: Two Studies of Mini-publics 
and Maxi-publics 


The parallel design from Example 1 was subsequently developed into a multi-level 
design in two new projects: one studying public support for alternative policies to 
tackle health inequalities;° and the other investigating the Citizens’ Assembly of 
Scotland about the future of the country.° What redefines these as multi-level designs 
is the addition of strands beyond the confines of the mini-publics. 


5 See http://www.healthinequalities.net (accessed March 1, 2021). 
6 See https://www.citizensassembly.scot (accessed March 1, 2021). 


400 Mixed Methods 


For example, the health inequalities project comprised three citizens’ juries in 
Manchester, Liverpool, and Glasgow, as well as a population survey that gener- 
ated data for the United Kingdom as well as for each jury location (see Smith et al. 
2021). This enables the comparison of deliberative versus aggregative constructions 
of ‘the public as a basis for analysing the resulting dilemmas for decision mak- 
ers facing alternative public-making approaches (Escobar et al. 2017). The MMR 
multi-level design thus helps to explore what types of publics are more supportive of 
upstream policies to tackle health inequalities (e.g. taxation, housing, labour, adver- 
tising) and which are more supportive of downstream policies (e.g. diet, smoking, 
fitness, lifestyle). These are very different policy responses to a complex challenge, 
and we are currently working on a paper outlining the implications for policymaking. 

The second example of a multi-level design is the research project about the Citi- 
zens Assembly of Scotland. Besides studying the internal dimensions of the Assem- 
bly (i.e. inclusion, design, facilitation, deliberative quality, governance), the research 
also considers its external dimensions. Therefore, it includes population surveys to 
investigate how the maxi-public relates to the mini-public and the issues undergoing 
public deliberation. This is complemented by interviews with institutional, political, 
and media actors to understand the systemic uptake and future prospects for demo- 
cratic innovation in Scotland. The research was recently completed (see Elstub et al. 
2022)’ and we have turned the shareable parts of the mixed methods dataset into 
an open access resource® so that interested researchers can work with it. The project 
illustrates the resource-intensive nature of MMR, in this case comprising a team of 
ten researchers across various disciplines. 


Example 3. Sequential Design: Studying the World 
of Official Deliberative Practitioners 


The final example is a sequential design to study the work of public engagement 
officials who organize and facilitate deliberative processes in local and regional gover- 
nance in Scotland. The first stage entailed two years of ethnographic fieldwork in four 
case study areas to develop a qualitative dataset including documents, images, partici- 
pant observation fieldnotes (i.e. 117 meetings, 131 days of shadowing, fifteen weeks of 
work placements), and transcripts from forty-four interviews and three focus groups 
(see Escobar 2014, 2015). The qualitative strand explored three questions: How do 
public engagement officials design and facilitate deliberative processes? What kind 
of work does it take? And what kind of work does this do (i.e. what is the impact on 
institutional cultures)? 


7 Interim data briefings are also available, covering internal dimensions to inform ongoing work by the 
Assembly’s Stewarding Group and Secretariat, as well as the design and facilitation teams: https://www. 
citizensassembly.scot/research (accessed March 1, 2021). 

ê The dataset has been deposited with the UK Data Service (https://ukdataservice.ac.uk/find-data/) and 
was being prepared for open access at the time this book went to print. 
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The findings provided the foundation to later develop a quantitative strand, thus 
turning the project into a sequential design that widened the inquiry. The rationale 
for MMR was thus threefold: development, complementarity, and expansion. Build- 
ing on the qualitative findings, a two-wave survey was conducted in 2016 and 2018 
to investigate the broader network of official engagement practitioners across Scot- 
land (Escobar et al. 2018; Weakley and Escobar 2018). The first wave provided a 
baseline, while the second also explored the early impact of the Community Empow- 
erment Act—new legislation introduced in 2015 to advance participatory governance 
and deliberative policymaking. The MMR sequential design therefore entailed an in- 
depth study of everyday work by a small group of practitioners, which generated 
ethnographically informed propositions to be tested across a larger population. This 
enabled the national surveys to be based on a grounded understanding of public 
engagement work at the frontline of deliberative processes. 

Crucially, MMR allowed us to check whether findings from the case study areas 
in the qualitative strand had resonance across the country (see Escobar et al. 2018; 
Weakley and Escobar 2018). For example, the surveys showed that levels of burnout 
amongst deliberative practitioners varied across localities, which tempered the stark 
findings from the qualitative strand (i.e. that burnout was rife). Conversely, we 
thought that the qualitative strand had focused on officials who were unusually 
committed to advancing culture change in public administration, and that there- 
fore these case study areas were outliers. The surveys, however, suggested that 
culture change work was prominent across the country, thus giving us insight into 
widespread ‘internal activism’ by public officials—a dimension that remains under- 
explored in democratic innovation (Escobar 2017a). All in all, this sequential MMR 
project spanned a decade and provided the evidence base to support meta-inferences 
about the current institutionalization of participatory governance in Scotland 
(Escobar 2021). 


Conclusion 


This chapter offered an overview of MMR, outlining its philosophical foundations, 
varied designs, strategies for analysis and integration, specialist terminology, and 
quality standards. I hope the chapter reads like an invitation to develop a mixed 
methods community of practice within the deliberative democracy community of 
inquiry. 

As we seek to enhance our current approaches in order to grapple with complex 
phenomena, we may grow in appreciation of what MMR has to offer. This may be par- 
ticularly so as deliberative democracy enters mainstream politics and policymaking 
and we try to understand, for example, the successes and failures of institutionalizing 
deliberative public engagement; how power dynamics unfold in deliberative systems; 
the work of policy actors and communities of practice in the deliberative industry; 
the prospects for public deliberation in everyday talk and multi-media contexts; how 
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various publics, stakeholders, and gatekeepers relate to citizen-centred deliberative 
institutions; or how online deliberation may enable transnational governance or the 
formation of a global demos to tackle the challenges of our time. 

MMR can foster collaboration across disciplines and temper the hegemonic ten- 
dencies of some research communities wedded to narrow definitions of scientific 
inquiry. All in all, MMR provides fertile ground for building an actionable science of 
and for democracy. Our current collective predicament, as citizens and researchers, 
demands no less. 


Further Reading 


Hesse-Biber, Sharlene Nagy, and Burke Johnson, eds. 2016. The Oxford Handbook of 
Multimethod and Mixed Methods Research Inquiry. New York: Oxford University 
Press. 

Teddlie, Charles, and Abbas Tashakkori. 2009. Foundations of Mixed Methods 
Research: Integrating Quantitative and Qualitative Approaches in the Social and 
Behavioral Sciences. London: Sage. 
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Case Study Research 


Stephen Elstub and Gianfranco Pomatto 


Deliberative democrats are usually interested in finding evidence for different kinds 
of causal claims. For example, what causes people to change their mind? In which 
contexts is deliberation most likely to cause opinion change? What conditions are 
required for good quality deliberation? Under what conditions can deliberation 
be consequential? There are different ways of studying these questions. Statistical 
research usually focuses on responding to these questions across a large number of 
cases, paying particular attention to average effects. 

There is a different way to answer these questions, however, and that is through 
case study research. Case study research can be defined as ‘the study of the particu- 
larity and complexity of a single case, coming to understand its activity in important 
circumstances’ (Stake 1995, ix). A case can bea person, event, activity, process, group, 
organization, or institution. The focus on an individual unit means that within- 
case analysis is conducted as opposed to comparative cross-case research (Ryan 
2019; see also Ryan, Chapter 17 in this volume). This is to enable intensive and 
context-dependent data collection and analysis on important cases that include a 
large number of variables, thereby producing more depth, detail, richness, complete- 
ness, and variance than cross unit analysis can generally achieve. In general, the more 
cases there are, the less data is collected on each (Gomm et al. 2004). Case stud- 
ies enable researchers to move beyond covariation between variables and focus on 
‘causes-of-effects’ in a specific case (Mahoney and Goertz 2006). 

Case study research has been prevalent throughout the history of deliberative 
democracy due to the depth of insight and attention to context that this method pro- 
vides (Boswell and Corbett 2017). In this chapter, we argue that case study research 
has been integral in helping deliberative democracy develop from a new normative 
theory to an established and mature theory guided by significant empirical evidence. 
Moreover, we suggest that as deliberative democracy research moves towards a sys- 
temic approach, case studies will continue to be essential for understanding the 
practice of deliberative democracy across the huge variety of contexts that a sys- 
tem involves. Case studies are a research approach that is relevant to both positivist 
and interpretivist research traditions. However, in this chapter, we suggest that case 
study research can help deliberative democracy move beyond this distinction. Guid- 
ance is provided on how to select a suitable case and how to collect and analyse data, 
specifically in relation to the field of deliberative democracy. 
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The Value of Case Studies in Deliberative Democracy 
Research 


Case study research is important to both the positivist and interpretivist research 
traditions. Both approaches are prevalent within deliberative democratic research. 
Nevertheless, these two broad methodological approaches utilize case study research 
differently for the purposes of responding to different research questions. The pos- 
itivist tradition is more interested in questions of generalization, which involves 
hypothesis generation and testing through the consideration of intervening variables. 
In contrast, the interpretivist tradition favours contextual richness, which includes 
considering the perceptions, values, and expectations of actors (Ercan et al. 2017). 
Here, cases are seen as specific and unique, worthy of understanding in their own 
right. We advocate a ‘third way, believing case study research has relevance to both 
traditions. 

Case studies are useful for building theory by generating hypotheses (Flyvbjerg 
2006; Crasnow 2012; Yin 2014; Toshkov 2016). The research identifies a phenomenon 
in one case and this leads to the generation of hypotheses that this will be found in 
the whole population or some of the population. However, there is much more scope 
for theory building in newer theories that are not yet established (Toshkov 2016). The 
use of case studies for theory building is often considered most appropriate within the 
positivist research tradition. However, deliberative democracy, emerging as it did in 
the late 1980s, is still a relatively new theory, so there has been plenty of scope for the- 
ory building and much of the case analysis has followed an interpretivist approach. 
For example, at the birth of deliberative democracy (Floridia 2014), we see seminal 
case studies on town hall meetings and cooperative workplace democracy in America 
using ethnography and qualitative interviews generating hypotheses, amongst oth- 
ers: that participatory processes can lead to representational and elitist dynamics; 
that people can find deliberation hard; and that features of complexity represent 
unavoidable facts, leading to conflicts of interests (Mansbridge 1983). Mansbridge 
(1983) concluded that participatory and deliberative approaches are more success- 
ful in decentralized processes, but must still be combined with more traditional 
representative and adversarial institutions. 

Deliberative democracy is a normative theory; it is about a counter-factual, polit- 
ical ideal. As a result, relevant cases of it being even approximated in practice have 
been few and far between. Particularly in its infancy, there were simply not the num- 
ber of cases available for large statistical or comparative research. As a result, case 
studies were often the focus of deliberative democratic research because they were 
the first case of a certain type. Consider, for example, the first ever deliberative poll 
(Luskin et al. 2002), citizens’ assembly (Warren and Pearse 2008), and participatory 
budget (Abers 2000). These practices were used as cases to investigate the extent 
to which ordinary citizens can deliberate about complex issues effectively, and to 
discern what institutional conditions are required to promote this. 
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This explains why case studies have been at the vanguard of empirical deliberative 
democracy research to date, but it is becoming an established theory now with devel- 
oped hypotheses that need testing and refining (Curato et al. 2017) and as a result we 
are seeing more positivist approaches that compare cases. Comparative content anal- 
ysis on the deliberative quality of cases, such as legislative debates (Steiner et al. 2004) 
and televised election debates (Marien et al. 2019), are some examples. Case study 
research is often criticized from a positivist perspective, claiming, for example, that 
it is impossible to generalize and to test theories as this would assume the full pop- 
ulation of cases was homogenous and causes deterministic (Stake 1994; Yin 2014; 
Boswell and Corbett 2017). Consequently, the statistical significance of the single 
case is usually unknown. This can certainly be the case if there is heterogeneity in the 
population under study and the hypothesized effects are weak: “There is not much 
point in testing a weak causal relationship relevant for a heterogeneous population 
with a single-case study design because the result of the investigation, whatever it may 
be, would have very little import’ (Toshkov 2016, 290). We must, therefore, be very 
cautious about making inferences about the population of cases, as a whole, from 
case study research. Should case studies, therefore, be sidelined for large-N statistical 
and comparative methods in deliberative democratic research now the theory has 
matured? 

We believe that this critique is misplaced for three reasons. First, we do not always 
need to seek generalization if the case is significant in its own right. This is often 
how cases are viewed from an interpretivist perspective. As democratic innova- 
tion increases in practice, we are still witnessing a stream of new cases, relevant to 
deliberative democracy, that merit in-depth, within-case analysis. This is a crucial 
development, as case study research contributes to the building of deliberative theory 
in important respects. For example, Hendriks (2016) was able to develop the con- 
cept of ‘designed coupling’ in deliberative systems only through in-depth case study 
research. 

Second, in contrast to what critics assume, generalization can be possible from a 
single case study, making case study research compatible with a positivist research 
tradition. It very much depends on the rationale for case selection. For example, 
Michels (1959 [1911]) researched the German Social Democratic Party (SDP) in 
1911. He selected this case study because the SDP had sought to be an internally 
democratic party, and as such it should be the least likely to be oligarchical. The fact 
that Michels found that the party was oligarchical led him to conclude most others 
would be too, without researching these other cases.’ Furthermore, a single case study 
can test hypotheses, and thereby also lead to generalization, if it falsifies a theory, 
providing a case is suitably selected for its validity. The classic example being a single 
black swan disproving the claim that ‘all swans are white. We would then need only 
to find one example of an internally democratic organization to falsify Michels’ ‘iron 


While there are some problems with Michels’ analysis of the case data (see Elstub 2008, 188-191), the 
rationale for case selection was sound. 
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law of oligarchy.” This is because average cases will not always be the most revealing: 
‘Atypical or extreme cases often reveal more information because they activate more 
actors and more basic mechanisms in the situation studied’ (Flyvbjerg 2006, 229). 

Third, at present, the focus of both theoretical and empirical research on delib- 
erative democracy is moving towards (or returning to) a focus on systems, which 
requires analysing the connections and relationships between different parts of the 
system (Elstub et al. 2016). It has been argued that deliberative systems must be 
researched in context and that this requires interpretivist methods, where hypothe- 
ses are less relevant, to understand the perspectives of a multiplicity of actors in 
numerous sites and the different types of communication that exist between them 
(Ercan et al. 2017). For example, case study analysis has developed our understand- 
ing of transmission in deliberative systems (Boswell et al. 2016). Case study research 
enables us to investigate a number of research questions on how instances of micro 
deliberation relate to the broader public sphere, such as how politicians view a par- 
ticular site or the discourses that emerge from it, or how citizens and associations 
view different political organizations and their discourses in particular contexts. 
These questions are not always possible to address through statistical analysis: ‘in 
large-N research it is not always possible to verify the incentives that the various 
actors face, the information they have and the beliefs they hold’ (Toshkov 2016, 299). 
Therefore, as deliberative democrats increasingly consider how deliberation can be 
scaled-up, the need for case study research grows. For example, how participants 
deliberate an issue, the likelihood of them changing their opinion, and their atti- 
tudes to deliberation, need to be researched in case studies embedded in real-life 
policy processes, precisely because they are imperfect, if we are to really understand 
deliberative democracy in practice. Communication about policy issues is inevitably 
different when a decision is going to be taken that will actually affect people, as 
indicated by a land management case study from the Peak District in the United 
Kingdom (Elstub 2010). Nevertheless, it is this type of communication that we must 
gain a greater understanding of if we are going to get greater leverage on deliberative 
systems in practice and revise the theory accordingly. 


Conducting Case Studies Research 


Case studies aim to extend our knowledge of how and why deliberative processes 
embedded in real-life policy processes work and do not work, as well as which ele- 
ments of the process lead to policy influence. To achieve this goal, case study research 
should critically examine the implementation of deliberative processes in both struc- 
tured forums and the unstructured public sphere; the dynamics between participants; 
and the broader social and political context in which the deliberative processes are 


* It should be noted that as a normative theory, deliberative democracy as a whole is not a falsifiable 
theory (Dryzek 2007); however, there are elements that could be refined through falsification. 
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embedded, including the reciprocal influences between the deliberative process and 
policy context. 

In other terms, interpretive case study analysis reconstructs the deliberative pro- 
cess as it has been implemented in the most accurate and detailed way possible. 
This enables us to make judgements as to their effectiveness from a deliberative per- 
spective because we examine how deliberative processes have been adapted to the 
external social and political context—the adjustments and variations that they have 
been subjected to during their implementation due to the explicit or implicit choices 
of the organizers, internal or external pressures, or other relevant factors. Differ- 
ent actors—organizers, participants, external influencers (such us political actors or 
the media)—should therefore be a key focus of case study research in deliberative 
processes. 

To achieve this, it is of paramount importance that case studies identify all the 
actors that, formally or informally, internally or externally, influence deliberative 
processes and their results. At the same time, it is crucial that case studies explain 
the purposes of the actors, the logic of action that they follow, the behaviour they 
adopt, the relations they establish with other actors, and their interpretation of the 
events (Scheirer and Griffith 1990; Pawson and Tilley 1997). Take, for example, the 
use of citizens’ juries as case studies on the reconfiguration of a hospital service in 
the city of Leicester in the UK (Parkinson 2006). The study not only focuses on the 
internal dimensions the jury, but also analyses the role played by all relevant external 
actors, suchas Health Authorities, the citizen committee that collected 150,000 signa- 
tures against a first version of the reconfiguration plan, health professionals, unions, 
charity groups, and the media. As highlighted by the author of the study: “The key 
feature of this case to keep in mind is that the jury was not the only element, but just 
the end point of a much bigger, Leicester-wide debate’ (Parkinson 2006, 14). 


Case Selection 


Choosing a case to study is the first step in case study research. Yin’s (2014) distinction 
between four main types of case studies with different rationales for selection, albeit 
not exhaustive, is widely used by empirical scholars undertaking case study research: 


1. The revelatory case is one that reveals new or underexplored phenomena. In the 
context of deliberative democratic research, the revelatory case enables access 
to deliberative processes on which existing literature is not available at all or 
is largely absent. In this situation, case studies have an explorative aim; they 
are useful to produce empirical evidence of what deliberative processes con- 
sist in and to elaborate inductive hypotheses to be tested in further research. 
Revelatory case studies can be carried out in innovative or unprecedented 
applications of the deliberative approach, as in the case of the deliberative pro- 
cesses on constitutional reforms (Levy et al. 2018), or cases implemented in the 
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context of deeply divided societies (Steiner et al. 2017), such as the case study 
of a deliberative poll on education policy in Northern Ireland to demonstrate 
that citizens can deliberate constructively together in these contexts (Luskin 
et al. 2014). Although deliberative approaches are growing in number, they 
are also expanding in geographical and policy reach, variety, and significance. 
Therefore, there are many revelatory cases that merit further study. 

. The common case is one that is prevalent, typical, or representative. In deliber- 
ative democratic research, it might consist in a deliberative process presenting 
a typical design embedded in a sociopolitical context with ordinary charac- 
teristics. Mini-publics on salient issues in local communities characterized by 
average social conditions and deeply rooted democratic institutions belong to 
this category. One example is the deliberative processes implemented in the 
Tuscany Region in Italy, financed by a regional law on participation of citizens 
in policymaking approved in 2007. In the first five years of implementation, 116 
deliberative processes were funded. Most of them were processes promoted 
by centre-left or left-wing municipalities in small localities characterized by 
an active civil society. Participants were generally recruited through random 
selection and the topics under discussion were not particularly conflicting, 
focusing on issues such as urban renewal and land planning (Lewanski 2013). 
. The critical case presents conditions and contexts that are particularly challeng- 
ing or extreme. Examples of these kinds of cases could be deliberative processes 
implemented on particularly controversial issues or in a context in which 
democratic institutions are weak or even non-existent. Case studies of deliber- 
ative democracy in China, for example, include deliberative polling (Leib and 
He 2006); deliberation in rural and urban areas and in a state-owned enter- 
prise (Unger et al. 2014); and online forums (Medaglia and Yang 2017). The 
implementation of deliberative processes in China is a critical case of particu- 
lar interest because it consists in an unexpected and apparently contradictory 
combination of an authoritarian regime at the national level and deliberative 
practices at the local level. The study of several empirical cases of local deliber- 
ative processes suggests that deliberation and democracy are not synonymous 
and that a deliberative authoritarianism could develop. 

. The unusual case can be considered a variation of the critical case. It con- 
sists in a deliberative process particularly rare or uncommon for its design 
or for the context in which it is embedded, albeit not particularly adverse or 
challenging. The study of unusual cases is useful in shedding light on specific 
factors influencing deliberative processes that are poorly studied so far. For 
example, Davidson et al. (2017) analysed the deliberative quality of the first 
ever televised election debates in the UK. The Ostbelgien Model in Belgium 
represents an unusual case since it is the first permanent citizens’ assembly 
(Niessen and Reuchamps 2019). Unusual cases are different from revelatory 
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cases as the latter explore new or underexplored phenomena, but are not nec- 
essarily rare cases. The unusual case is always a rare case, but it does not 
necessarily pertain to a new or underexplored phenomenon. 


Data Collection 


The second step to conduct case studies consists in collecting empirical data in the 
field. It is beyond the scope of this chapter to cover the different methods of data 
collection that can be used in case study research in detail. Most of the methods pre- 
sented in this volume are suitable for case studies. Importantly, data can be collected 
which illuminates the deliberative process itself, as well as its relationships to other 
actors and institutions, and causes of the outcomes. Therefore, the aim is to collect 
a large amount of data on the case to analyse a significant number of variables or 
dimensions. Nevertheless, data collection choices inevitably need to be made, even 
when researching just one case. The data-gathering strategy should be determined by 
the nature of the research question and the type of case study that has been selected. 
However, it is worth noting that case studies are particularly suitable for mixed meth- 
ods approaches (Escobar and Thompson 2019; see also Escobar, Chapter 27 in this 
volume). This can enable triangulation—the application of several research methods 
to study the same phenomenon—but also increases the amount of evidence we have 
to assess causal hypotheses (Gerring 2004) or to enhance the interpretation of the 
case. For example, the mixed method study on the case of citizens’ juries on onshore 
windfarms in Scotland had several data sources including juror survey data, non- 
participant observation, interviews with the witnesses, and an analysis of the jury 
discussions, which enabled triangulation about good practice for the use of evidence 
in mini-publics (Drury et al. 2021; Roberts et al. 2020). 

Relevant quantitative data collection can include characteristics of the processes 
under investigation, for example the number of participants, duration of process, vol- 
ume of media coverage, etc.; surveys of participants that can help evaluate opinion 
change in the process and attitudes towards the process; and content analysis that can 
test the deliberative quality and media coverage of the case. However, these aspects 
of one case are not comparable in the same way that we might compare a num- 
ber of cases on just one variable (Toshkov 2016). Relevant qualitative data include 
descriptions, narratives, opinions, and interpretations on what has happened during 
the implementation of the processes and after their conclusion, from those involved 
and other stakeholders. This can be collected through archival research, participant 
observation, interviews, and focus groups involving actors and other key informants, 
often in combination. Secondary literature can be used before collecting evidence 
from primary sources in order to have an overall picture of existing information 
and interpretations. Moreover, it can be used to enrich and problematize evidence 
deriving from primary sources. 
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The third step to conduct case studies is analysing the data in order to produce a 
coherent, plausible, and well-founded interpretation of the specific social mecha- 
nisms working in the case under investigation. There are two main challenges here. 
The first risk is that of only producing a detailed chronicle of the studied process that 
does not go beyond the descriptive level. The second risk is the opposite: producing a 
subjective interpretation of the processes based on cherry-picking and in line with a 
preconceived thesis. Two main recommendations are useful in seeking to avoid these 
risks and they apply to all types of case studies. 

The first recommendation is to avoid a simplistic interpretation of the behaviour 
of social actors. For example, exclusively referring to rational choice theory where 
social actors would be considered to be motivated only by egoistic interest would 
result in a biased narrative with actors depicted as caricatures. Take, for example, a 
case study on Tuscany law on citizens’ participation in policymaking (O’Miel 2016). 
The case study focuses on the role played by some actors that directly contributed to 
the formulation of the law: a politician, a scholar, a civil servant, and a practitioner. 
All of them are depicted as being motivated by self-interest. The analysis suggests 
that they contributed to the formulation of the law in order to extend consensus, 
to achieve legitimation, to strengthen their own influence, and to obtain economic 
advantages. In contrast, Bobbio and Floridia (2016) highlight that a major role in 
fostering a new regional law on citizens’ participation in Tuscany was played bya large 
movement of associations and intellectuals, and that, therefore, the role attributed to 
the presumed self-interest of a limited number of actors, albeit influential, is a clear 
deformation of what transpired. The authors highlight that to produce a profound 
and realistic interpretation, researchers should refer to a plurality of social theories: 
the behaviour of actors is influenced not only by interests, but also by values, beliefs, 
cognitive shortcomings, and shortcuts, the social and historical context in which they 
are embedded, and the networks of which they are a part, amongst other factors. 
Therefore, researchers should exercise a critical analysis in all possible directions, 
taking into consideration, comparing, and questioning competing hypotheses. 

The second recommendation is to conduct the case study as a skilled detective 
solving a crime mystery. In this analogy, hypotheses are suspects. A detective col- 
lects clues in the field, identifies all the possible suspects with different motives and 
analyses, and combines the clues until a solid proof of the guilt of one of the sus- 
pects is reached. Data analysis is not sequential, but, developed along an iterative 
and incremental process. During this process it is crucial for the researcher-detective 
to evaluate the solidity of hypotheses in the light of the collected evidence. Process 
tracing methodology, also called causal process observation, is a method of qualitative 
analysis that is particularly useful for case study analysis (see Pickering, Chapter 20 of 
this volume). It can be defined as ‘tracing the decision process by which various initial 
conditions are translated into outcomes’ (George and McKeown 1985, 35). Given the 
focus on decision-making, it is a useful approach for research on the policy processes 
and institutions from a deliberative democratic perspective, because it provides a 
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clear formalization of the logical steps that a detective-researcher has to take in order 
to reach a valid interpretation. These logical steps can be considered a tool in the 
hand of researchers in order to avoid simplistic and biased interpretations, and helps 
to explain why the proposed interpretation has been developed. 

Process tracing provides four logical tests of causal hypotheses and is useful for 
evaluating different pieces of evidence and their contribution to causal inference and 
whether they are necessary and/or sufficient for the cause to occur (Collier 2011): 


e The straw-in-the-wind test (which way is the wind blowing?): A hypothesis 
passes this test when some of the collected evidence could be effectively con- 
sidered to be supporting it, but at the same time the evidence is not decisive. 
It is useful for the researcher to conduct an initial assessment of the collected 
evidence and to better focus on which directions merit further investigation. 

¢ The hoop test (jumping through the hoop): The collected evidence is useful to 
eliminate one or more hypotheses and to strengthen one or more other hypothe- 
ses. This means that the researcher is getting closer to a plausible interpretation, 
but other efforts are yet required. 

e The smoking-gun test (a murder suspect is found holding a smoking gun): If 

a hypothesis passes this test, then it has very strong support. The alternative 

hypotheses are substantially weakened, although not completely eliminated. 

The doubly decisive test: If a hypothesis passes the test, it is confirmed and at 

the same time all the alternative hypotheses are rejected. It is a level of solidity 

that is particularly difficult to achieve, and it can be considered an ideal research 


objective. 


The tests vary according to how unique they are and their level of certainty. The 
former relates to how likely it is that the evidence will be there if the hypothesis 
is not true, and the latter if it is true: “High certitude means that the hypothesis is 
very unlikely to hold if the piece of evidence is not available and vice versa . . . High 
uniqueness means that, if the evidence is found, it could have been produced under 
only one hypothesis and not others. Low uniqueness means that the evidence could 
have been left by several competing hypotheses’ (Toshkov 2016, 295). How these four 
tests relate to the two criteria is depicted in Table 28.1 below. 

A useful way to highlight the validity of this approach for research on deliberative 
democracy is the case study on a public debate on the project of a highway in Italy that 
has combined participant observation with in-depth interviews with key informants 


Table 28.1 Types of Evidence and Their Implications 


Certitude 
High Low 
Uniqueness High Doubly decisive Smoking gun 
Low Hoop Straw-in-the-wind 


Source: Toshkov (2016). 
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(Pomatto 2015). The conflict between citizen committees and public institutions will- 
ing to build a new highway lasted more than ten years. Before the beginning of the 
debate, citizen committees were particularly distrustful of it. They maintained that 
the public debate was an attempt by public institutions to produce consensus for the 
infrastructure without any real possibility to change the original project. A failure of 
the public debate and the prosecution of the conflict between citizen committees and 
public institutions without any changes appeared probable. However, the study high- 
lights that public debate did not fail and that some positive solutions can be reached 
even if the conflict is not completely resolved. It is a thesis progressively developed 
by researchers in the field using the logical tests provided by the Process Tracing 
Methodology. 

At the beginning of the debate, citizens’ committees, which were strongly against 
the infrastructure under discussion, accepted the invitation to take part in the public 
debate. This is a first piece of evidence that passes the straw-in-the-wind test. On the 
one hand, the fact that citizen committees did not boycott the debate does not mean 
that they were willing to discuss the project with the proponent of the highway. On the 
other hand, it suggests that the conflict is not completely unresolvable, meaning that 
there is merit in further research on the behaviour of the citizen committee during 
the debate. 

Two revelatory episodes occurred in two different public meetings of the debate. 
On both occasions, the facilitator did not tolerate contestations from a group of par- 
ticipants against supporters of the highway who were speaking. In both instances, the 
facilitator was strongly challenged by the participants and in both cases the leaders 
of the citizen committees stopped the contestation against the facilitator and asked 
the public to be respectful. These two episodes constitute evidence that passes the 
hoop test. They strengthen the hypothesis that citizen committees wanted the debate 
to continue. However, these episodes do not prove that positive sum solutions can 
be produced through the debate. 

In the last part of the debate, two lay citizens advanced some proposals to change 
the layout of the highway in order to reduce the project’s negative externalities. These 
citizens discussed their proposals with technicians in a specific workshop. The new 
project advanced by the technicians after the conclusion of the debate included a 
variation of the original layout very similar to the proposals advanced by these cit- 
izens. This is evidence that passes the smoking-gun test: it proves that the public 
debate could produce positive sum solutions even if it is implemented in a strongly 
conflicting context (Ravazzi and Pomatto 2014; Pomatto 2015). 


Conclusion 


In this chapter, we provided an overview of how case studies can be used in delib- 
erative democratic research. Case study research is not always required, nor is it 
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always the best approach. Far from it. Comparison across cases, experiments, and 
large random sample research methods are hugely valuable for the study of deliber- 
ative democracy: “The advantage of large samples is breadth, whereas their problem 
is one of depth. For the case study, the situation is the reverse. Both approaches 
are necessary for a sound development of social science’ (Flyvbjerg 2006, 241). Our 
epistemological approach does not lead us to favour a particular methodological tra- 
dition. Rather, we see methods as tools and it is a matter of selecting the right tool 
for the job in hand, so it very much depends on the research question. Case studies 
are very useful for investigating causes in context, and should therefore continue to 
be a method widely used for deliberative democratic research. 
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ENACTING DELIBERATION 


29 
Deliberative Policy Analysis 


Hendrik Wagenaar 


It has been almost two decades since Maarten Hajer and Hendrik Wagenaar 
published Deliberative Policy Analysis (hereafter DPA). The book’s publication 
was considered a major development in the post-positivist policy movement. 
The book’s subtitle— Understanding Governance in the Network Society—indicated 
DPA’ programmatic approach. It argued that the changing nature of the political- 
administrative system made post hoc, research-based information less effective as 
input into processes of political decision-making. The editors, in their introduction 
to the book, depicted these changes in terms of a ‘network society’. In today’s terms, 
the vocabulary would be one of complexity (Wagenaar 2007; Gerrits 2012), turbu- 
lence (Ansell and Trondal 2018), or uncertainty (Funtowicz and Ravetz 1993). This 
led the editors to ask: “What kind of policy analysis might be relevant to understand- 
ing governance in the emerging network society?’ and to posit a lack of fit between 
dominant, positivist, technocratic forms of policy analysis and the predicament of 
political and administrative decision-making (Hajer and Wagenaar 2003, 13). The 
editors have argued that for policy analysis in the network society to be effective 
and democratically legitimate, it should be interpretive, practice-oriented, and delib- 
erative. That is, instead of the traditional after-the-fact academic policy study, the 
editors of DPA proposed that policy analysis should be forward-looking and situated; 
it should involve stakeholders in problem setting and solution design; and it should 
capitalize on their practical, experiential knowledge. DPA aimed—once again—to 
make good on Lasswell’s historic call for a policy science of democracy (Lasswell 
1971; Dryzek 1989). 

Since the book’s publication, the development and uptake of DPA has been both 
inspiring and frustrating. Many readers found the book’s diagnosis of the limits of 
technocratic policy analysis convincing. They also regarded the book’s central mes- 
sage persuasive, namely that an interpretive and participative form of policy analysis 
is better equipped to address the challenges that the dynamic, interconnected nature 
of contemporary society poses to policymakers. However, as Li observes (2019), the 
absence of a clearly recognizable methodological approach, a set of operational and 
replicable procedures that potential practitioners can make their own, has hampered 
the diffusion of DPA. People who are, in principle, sympathetic towards DPA have a 
hard time figuring out how to actually do it. 


Hendrik Wagenaar, Deliberative Policy Analysis. In: Research Methods in Deliberative Democracy. Edited by Selen A. Ercan, 
Hans Asenbaum, Nicole Curato, and Ricardo F. Mendonca. Oxford University Press. © Oxford University Press (2022). 
DOI: 10.1093/0s0/9780192848925.003.0029 
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In this chapter, I offer three methodological approaches to DPA. I have articulated 
them by inductively drawing on the recent literature on applications of DPA in var- 
ious settings and countries (Li and Wagenaar 2019; Bartels et al. 2020b). I speak of 
‘methodological approach’ and not ‘methods’ because of the open-ended nature of 
DPA. In keeping with the complex, indeterminate character of the policy world that 
DPA attempts to harness (Axelrod and Cohen 2000), I argue that the very purpose 
of DPA is to maintain a flexible, relational, open-ended approach to policy analysis.’ 
In addition, one of the key insights of DPA is that it is more than just an epistemic 
innovation. By striving for institutional transformation, it collapses the boundaries 
between policymaking and policy analysis, claiming that the two are continuous. This 
does not accord with rigid methods. DPA does not operate like an algorithm but more 
like a heuristic, tailored to specific situations. The term ‘heuristic denotes a strategy 
of discovery (Abbott 2004, 81), or, in different terms, a strategy of abduction (Tavory 
and Timmermans 2014). 

Abduction is a kind of reasoning that ‘produces plausible provisional results— 
insights, guesses and concepts that link things together in new ways ... Abductive 
reasoning produces insights and ideas that are plausible but provisional’ (Kimbell 
2015, 35). Abduction results in conceptual understandings and practical solutions. 
It is a central element of contemporary ‘design-in-practice, which is not rationalist 
and comprehensive but emergent (Kimbell 2012). It is interactive and collabo- 
rative, using a variety of insights, experiences, and material artefacts to link the 
desirable, the feasible, and the viable (Kimbell 2012, 294). Design-in-practice has 
articulated a forward-looking ‘epistemology of creative work using a repertoire of 
resources (from relational reconfiguration to organizational change), to generate new 
insights, solutions, and practices, as well as organizational structures and institutional 
arrangements, in situations of indeterminacy and uncertainty (Kimbell 2012, 295; 
see also Goodin 1996, 29). Emergent, interactive, practice-based design is a central 
feature of DPA methodology.” 

It is thus possible to formulate three distinct methodological approaches to DPA: 
(1) Designed Deliberative Forums; (2) the Enhancement of the Deliberative Capacity 
of a Policy System; and (3) Institutional Design-in-Practice. These three approaches 
do not exhaust the full range of DPA-like methods. I suggest, however, that other, 
related methods such as co-production (West et al. 2019) and action research 
(Bartels and Wittmayer 2018; Bartels et al. 2020a) show family resemblances to 


1 There is an important literature on relational ontology in various disciplines that is relevant to DPA 
but that, for reasons of space, can only be mentioned here. Relational ontology ‘assumes that what primar- 
ily exists is relation, not entities like things and individual human beings’ (Sidorkin 2002, 91). Relational 
ontology has its roots in the pragmatism of Mary Parker Follett and John Dewey, and the process philos- 
ophy of Alfred North Whitehead. It has been further developed and applied in anthropology (Sidorkin 
2002), sociology (Emirbeyer 1997), and recently in Public Administration and policy studies (Stout and 
Love 2018). Through its relational roots, DPA has a strong family resemblance with Action Research 
(Bartels and Wittmayer 2018; Bartels et al. 2020). 

° After all, Deliberative Policy Analysis was published in Cambridge University Press’s Theories of 
Institutional Design Series. 
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these three approaches. In fact, they all share transformative ambitions through an 
interventionist, ‘(actionable) methodology of co-producing inquiry and social inter- 
vention with all relevant stakeholders’ (Li and Wagenaar 2019, 432). This range of 
methods shows that DPA is a versatile approach to policy analysis that covers dif- 
ferent scales and modes of the policymaking process and is useful to policymakers, 
stakeholders, social activists, and researchers alike.’ 


Designed Deliberative Platforms 


The most frequently used methodological approach to DPA is Designed Deliberative 
Platforms (DDPs). DDPs come under various labels such as collaborative governance 
(Ansell and Gash 2008), collaborative planning (Innes and Booher 2010), interactive 
governance (Edelenbos and van Meerkerk 2016), or policy labs (Kimbell 2015). All 
these approaches have in common that the policy process has broadened to include 
deliberation with stakeholders and citizens in the problem formulation and design 
stage of the policy process.* 

Designed Deliberative Platforms show several recurrent elements. First, they are 
problem-oriented. They are organized around a concrete policy issue. Bowman, in 
a paper on the pragmatist roots of DPA, argues that an orientation towards resolv- 
ing concrete problems that originate in doubt, ‘promoted by a problematic situation’ 
in the pragmatist vocabulary, is a necessary condition for all inquiry (2019, 560). 
And Ansell, in his pragmatist-informed treatise on democratic governance, con- 
siders a ‘problem-driven perspective’ to be a prerequisite of ‘evolutionary learning. 
‘[P]roblems disrupt existing assumptions and call for fresh recovery, he says. “They 
pin disputes about knowledge, principles, and values down to particulars . . . and they 
focus our attention on action and consequences’ (2011, 11). 

Second, deliberation is a strategy of policy inquiry. The cases testify to what prag- 
matists call ‘the communal basis of inquiry’ (Ansell 2011, 12). Deliberation aims at 
discovering shared goals and values, recognizing interdependence, and articulating 
shared problem definitions. These cognitive goals (reflection, reframing, and evo- 
lutionary learning) are arrived at through establishing new relationships—one of 
the major strategies of harnessing complexity. For example, Forester and colleagues 
speak of ‘establishing new pathways’ for forging relationships and agreeing on prob- 
lem definitions and discovering new, creative solutions (Forester et al. 2019, 468). 
They talk of ‘relational re-braiding’ and describe it as follows: 


> In their book ‘The Pandemic Within: Policy making for a Better World’ (2021), Wagenaar and Prainsack 
introduce utopian re-imagination as a fourth, systematic, practice-oriented method of deliberative policy 
analysis. It exceeds the limits of this chapter to explain this method in detail. I refer to the book for further 
details. 

* Lis Laboratory of Deliberative Policy Analysis in Beijing—in which citizens and experts engage in 
guided deliberation about contested issues in a kind of citizen jury-type arrangement, the results of which 
are subsequently presented to the authorities—is a good example of this approach (Li 2015). 
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Here we have a quality of creating not just a view of the past, but a lived present in 
which participants who have not been in the same room together slowly ‘met’ each 
other, slowly came to develop a new trust in one another, slowly came to see each 
other not just vaguely ‘in new ways’, but with new qualities of respect, apprecia- 
tion, recognition, acknowledgment, curiosity, empathy, and mutual regard. These 
are the deliberative accomplishments, the relational accomplishments. 

(Forester et al. 2019, 469) 


In complexity terms, changing the relations in a group or policy system amounts 
to facilitating nonuniform interaction patterns (Axelrod and Cohen 2000, 63). Out- 
comes of the system arise from the complex interactions of its material, human, 
and symbolic agents. Effectuating change in both the proximity of interactions (who 
interacts with whom) and activation patterns (the sequencing of interactions) can 
have profound effects on the problem-solving capacity of the system. The change 
mechanisms are multiple. Increasing diversity opens the system up to new values, 
knowledge, perspectives, skills, and ideas. Diversifying interaction patterns also trig- 
gers different co-evolutionary patterns that, in the long run, may change the system 
(Axelrod and Cohen 2000, 68-69). Changing interaction patterns through delibera- 
tion is a practice. This is never easy in conflictual situations, but accomplishing this 
task results in recognition of mutual dependence, joint ownership of the problem, 
while new professional and personal relationships set in motion reflective processes 
that facilitate mutual learning (Axelrod and Cohen 2000, 38; Ansell 2011, 177-178). 
The relational dimension of deliberative inquiry results in a deeper form of lived, 
embodied learning—learning that sticks. 

Third, DDPs emphasize the continuity of inquiry and action, where practices drive 
the process of joint inquiry. Deliberation in the service of practice is coterminous 
with policy design as a ‘distributed social accomplishment in which artifacts and 
other humans play an important role’ (Kimbell 2011, 300). Practice is an answer to 
uncertainty. As Cook and Wagenaar state: 


[W]e live in a world where problem formulations are often unclear and contested, 
where it is unclear what the materialities of the situation will ‘afford’, where the 
implications of alternative actions are uncertain, and where the utility of various 
kinds of knowledge is not immediately apparent. 

(Cook and Wagenaar 2012, 17) 


Therefore, from a practice perspective, the initial task of the actors involved in a trans- 
disciplinary research project is not to apply knowledge but to devise a productive 
form of practice within which the problem can be addressed. Acting or ‘doing’ is cen- 
tral here, from interpersonal dialogue to physical tinkering and interaction with the 
environment, to ‘get the measure of the situation and figure out ‘how to go on in light 
of affordances and constraints, purposes and expectations (West et al. 2019). A good 
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example is administrative discretion. Because formal rules are inherently limited in 
the extent to which they are able to shape behaviour, to make administration work 
at all requires practical engagement with the situation at hand and the professionals 
and clients that populate it. Out of this practical engagement, understanding, moral 
judgment, and decision-making emerge (Wagenaar 2020). A practice approach to 
policy analysis fuses analysis, experience, and interaction. 

Fourth, visual tools are used to explicate both functional and dysfunctional policy 
processes. Visual tools such as timelines or diagrams, although deceptively simple, 
serve performative and relational functions in design-in-practice (Kimbell 2011, 
237). They are crucial for getting perspective on a conflict when participants are 
immersed in their own jealously guarded sense of rightness. They suggest avenues for 
a resolution of the conflict, in this way breaking the impasse. The key to this is that 
the visual aids are constructed jointly. For example, the central element of the recon- 
struction clinic described in Forester et al. (2019) is the collective construction of a 
timeline. Starting from a situation of impasse and entrenched positions held in place 
by strong emotions and ‘self-evident’ value positions, the timeline is a visual recon- 
struction of the chronology of the conflict based on events that are critical according 
to the participants, ‘with quotes from each party in a different color’ (Forester et al. 
2019, 461). With the help of two neutral commentators, so-called ‘reflectants’, the 
participants began to develop a ‘collective insight that the situation that they were in 
was something they were in together and that they [had] created together’. The time- 
line and the reflectant’ commentary helped the participants to become ‘unstuck’ It 
helped them both to see their different interests and modi operandi as well as their 
mutual dependency. Interviews with participants also showed that the construction 
of the timeline was a subtle form of empowerment. It allowed the weaker parties in 
the process, such as citizens, to be heard and to have an input in the process. The 
final step was to ask participants to jointly construct a future timeline. The outcome, 
according to the organizers, was that the conflicting parties began to see the situ- 
ation as a ‘collective mess that needed collective action in order to move forward. 
Individuals became part of a system and attributions to others morphed into shared 
responsibility’ (Forester et al. 2019, 462). 

Fifth, DPA invites participants to embrace complexity. The open, deliberative for- 
mat of DPA encourages receptiveness to surprise and improvisation. For example, 
Foster et al. (2019) are explicit in their approach to complexity. Their ‘systematic 
co-inquiry’ is designed as a methodical approach that begins with acknowledging 
uncertainty and ‘focuses on processes of social learning and the emergence of oppor- 
tunities, rather than on pre-defined timelines, blueprints and outputs common to 
projects and programs’ (Foster et al. 2019, 522). The authors used visual diagrams to 
depict both the messiness of the current situation and the more organized complexity 
of the UK Water Management System. Similarly, a “Future-Proofing’ rainforest con- 
servation initiative in Colombia grew out of ‘dissatisfaction around existing technical 
approaches to climate adaptation. 
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Strategies for tackling climate change were rooted in the linear assumptions of 
the traditional model—that more science would reduce uncertainty, and that less 
uncertainty would lead to decisive action in rethinking conservation governance 
to accommodate climate change. 

(West et al. 2019, 545) 


However, these institutional routines were shattered when ‘existing conservation 
interventions were no longer adequate, and the world was indeed “talking back” in 
increasingly strident tones. The team began to grapple with a situation of ‘ecolog- 
ical instability that many felt was certain to increase’ (West et al. 2019, 546). The 
ecologists involved in the conservation project settled on a pragmatist-inspired strat- 
egy of evolutionary learning that consisted of a series of interlinked elements. A key 
element was a shift from conservation thinking to one of anticipating and accom- 
modating change. As both the direction and extent of change in a complex system 
are, by definition, unpredictable, this required a shift in both the object as well as 
the strategy of conservation management. The researchers sought the substitution of 
things (species, ecosystems) to values ‘that are contestable and more overtly political 
as ‘guides for management’ (van Kerkhoff et al. 2019, 702). 

But this strategy could only be successful in a system of co-production in which all 
partners were able to assess the state of existing knowledge and evolving knowledge 
needs relevant to each partner’s specific situation. That meant accepting uncertainty, 
a willingness to value the hands-on experiential knowledge of field rangers as guides 
for systemic action, and not hiding behind the lack of scientific evidence as a reason 
for inaction (van Kerkhoff, personal communication). Through a series of work- 
shops, the group came up with a ‘product’ that they describe as ‘a “process”—a 
multi-stage suite of activities that engages participants in a series of deliberations 
around conservation, culminating in a dialogue event to connect the pieces. The 
authors describe this process as the establishment of a ‘practice’: ‘As a practice 
it is ongoing, deliberative and potentially transformative, framed by learning and 
dialogue rather than the application of technical solutions’ (West et al. 2019, 548). 

These quotations demonstrate both the acknowledgement of the complexity and 
uncertainty that policymakers were facing and the open-ended, jointly produced, 
improvisatory character of the deliberative strategy. Embracing complexity, in con- 
junction with the tactics discussed above, helps to transform ingrained institutional 
practices, facilitate evolutionary learning, and build institutional capacity. In this 
way, all DPA examples discussed so far are aimed at harnessing complexity (Axelrod 
and Cohen 2000) rather than reducing it. 


Enhancing the Deliberative Capacity of a Policy System 


The second methodological approach to DPA is enhancing the deliberative capacity 
of a policy system. Designed Deliberative Platforms are an important and relatively 
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easy to implement methodology for working through conflict-ridden policy issues 
with the affected actors. By diversifying the lines of communication and opening 
deliberations to hitherto neglected local knowledge, DDPs release creativity and 
harness complexity (Axelrod and Cohen 2000). However, as some authors have 
argued, the place of deliberation in policy analysis is not always immediately obvious 
(Bachtiger et al. 2010, 40; Papadopoulos 2012, 126). As a result, DDPs face several 
risks that might ultimately limit their reach and effectiveness. 

First, it is not always clear what deliberation is for when introduced into the 
policymaking process. While the intended programmatic effects of DDPs are well 
articulated by their proponents (knowledge enhancement, intersubjective mean- 
ing, shared problem-solving, awareness of mutual dependence), these forums serve 
different de facto functions in institutionalized policy systems, such as conflict res- 
olution, cooptation of political opponents, mustering political support, delaying 
unpopular decisions, overcoming veto-points, or deflecting public attention from 
unpopular programmes. Although such strategic political considerations are not nec- 
essarily negative, they are often at odds with whatever direct benefits DDPs might 
have. This risk is aggravated by the epistemological vulnerability of DPA in the 
current institutional environment. In an institutional setting that is in the sway 
of instrumental reason and command-and-control models of governing and pol- 
icymaking, interpretive methods and a focus on practices have less standing than 
knowledge that is purportedly based on formal scientific methods (Taylor 1995, 7). 

Second, these problems are part of a more general obstacle for DPA. Most policy- 
making institutions are not particularly receptive to deliberation when left to their 
own devices. It is probably for this reason (to avoid the difficult work of persuad- 
ing political decision makers to accept the outcomes of deliberative platforms) that 
the examples discussed so far all represent carefully designed forums that (mostly) 
operate outside the regular chains of command and channels of political communi- 
cation. More often than not, deliberative innovations in governance exist alongside 
hierarchical imposition or the continued privatization of public services, seriously 
constraining whatever positive effects they may have. This then raises the issue of 
how these atypical platforms of communication produce actionable, transformative 
knowledge that can be integrated into the everyday policy process. Differently put, 
it is ‘important to look at how forums play out in larger systems of governance, for 
it is the deliberative virtues of the latter that is ultimately the main concern (Dryzek 
2016, 231). 

The earlier mentioned Future-Proofing initiative in Colombia is an example of 
enhancing the deliberative capacity of a comprehensive policy system as a methodi- 
cal approach to DPA. The Future-Proofing Conservation project explored new ways 
of managing protected rainforest areas that were challenged by large-scale, rapid, 
destructive ecological change (van Kerkhoff et al. 2019). What makes the case inter- 
esting is that its design transcended the usual small-group deliberative initiative and 
focused on climate governance as a systemic issue. The team consisted of government 
officials, activists, academics, NGOs, and consultants who were dispersed over four 
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continents. In this sense the group was more akin to a deliberative system than a 
deliberative forum (Mansbridge et al. 2012). For example, while the focus of the 
project was on improving the uptake of knowledge in conservation policies, the actors 
realized that climate adaptation is not only a scientific but also, above all, a gover- 
nance issue. In fact, knowledge (of all kinds) and governance were seen as integrated 
elements of a general strategy of evolutionary learning (van Kerkhoff et al. 2019; 
Ansell 2011). Evolutionary learning is a form of institutional capacity building that 
embraces three principles: (1) a problem-driven perspective; (2) facilitating processes 
of practical judgement—or, in the words of the organizers, ‘supporting processes of 
reflexivity at both individual and institutional levels, thereby recognizing different 
possibilities for change and transformation, as well as constraints—recognizing the 
“policy mud”; and (3) creating ‘spaces for deliberation that include consideration of 
personal and social values, historical processes of change, and aspirations for pos- 
itive futures’ (West et al. 2019, 546). Throughout the process, the researchers kept 
their eyes on park conservation as a systemic issue. They were interested to learn, 
with partners, about the context of park conservation. ‘Participants appreciated that 
the process enables them to think deeply about the “bigger picture” of conservation, 
and to relate that to their more immediate management context’ (van Kerkhoff et al. 
2019, 10). 

The Future-Proofing initiative points the way to a feasible, design-driven, hands- 
on approach to enhancing the deliberative capacity of policy systems as a whole. 


[Bly developing this larger transition to new governance, through a series of 
smaller, interconnected transitions that linked values, rules and knowledge, the 
participants could work through a series of steps that enable new ways of thinking 
about the role of protected areas in conservation and consequently new ways to 
manage them. 

(van Kerkhoff et al. 2019, 13) 


Institutional Design-in-Practice 


Finally, the third methodological approach to DPA is Institutional Design- 
in-Practice. DPA claims to have an answer to complexity. It mobilizes intense small- 
group deliberation to bring conflicting parties together, unfreeze policy impasses, 
and set in motion design-in-practice. In the process, it forges new forms of col- 
laboration that unleash creative problem-solving. It has proven itself capable of 
liberating the wisdom of practice in, and enhance the deliberative capacity of, local 
and national policy systems to initiate evolutionary learning. But complexity intro- 
duces two qualities that have yet to be addressed so far: interconnectedness and 
higher-order effects. 
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First, through a myriad of continuously evolving relationships, the elements in 
a complex system are densely interconnected. Positive and negative feedback cre- 
ates the unexpected feedback loops and confounding emergent outcomes that are 
characteristic of complex systems. Interconnectedness poses real problems to policy- 
makers. We intervene knowing that our action will have consequences that reverber- 
ate through systems in unpredictable, often nonlinear, ways, expressing themselves 
as the absence of any desired effect or alternatively as negative or perverse unin- 
tended consequences (Sterman 2000; Wagenaar and Prainsack 2021, chapter 1). We 
also know that our models of the world that we use as leverage for our interventions 
are limited at best. Yet, if we ever hope to design policies that are effective, that mini- 
mize negative unintended consequences, and that are sufficiently robust to withstand 
changing circumstances, then our interventions need to be commensurate with the 
multiple interactions and dependencies between elements of the system. That means 
that the policymaker not only understands the compound nature of the problem at 
hand, but also designs a differentiated, integrated policy whose elements support and 
reinforce each other in attaining their goal. 

Second, the emergent outcomes of complex systems are a feature of the interactions 
between entities. The higher-order phenomena that emerge through these interac- 
tions have properties that cannot be reduced to those of lower-order phenomena. 
They cannot be reconstructed from the latter by simple extrapolation. The reason is 
that at each level of aggregation, properties appear that are specific to that level and 
that require their own concepts, explanations, and generalizations (Anderson 1972, 
393; Wagenaar 2007, 25). This has momentous implications for an interventionist, 
transformative form of policy inquiry. Interconnectedness and higher-order effects 
in policy systems go beyond interpersonal relations in that they involve aggregate 
phenomena, such as organizations, institutions, buildings, laws, regulation, policy 
instruments, as well as natural systems, in short, the stuff of public administration. 
To harness this kind of administrative and natural interconnectedness requires an 
ability to intervene intelligently in these higher-order entities. ‘Intervening intelli- 
gently’ is the language of design, in the case of DPA, the design of institutions of 
public administration and public policy. 

Understanding and accommodating interconnectedness requires a different level 
of analysis. Not on the elusive level of comprehensive systems, but rather a focus on 
institutional design where the leverage points are precisely the organizations, institu- 
tions, policy instruments, and professional roles that comprise policy systems. One 
of the goals of the Future-Proofing project, for example, was not only to bring indige- 
nous actors into the deliberative monitoring system, but also to strengthen relation- 
ships between local/regional and national levels of management (van Kerkhoff et al. 
2019). In addition, the researchers developed a Protected Area Benefits Assessment 
Tool that invited area managers to link landscape features with the benefits and costs 
of interventions in light of expected climate change (van Kerkhoff et al. 2019). To gain 
traction, laws and programmes have to be operationalized, by being translated into 
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a series of policy instruments, such as dedicated organizations, budgets, information 
tools, and monitoring and enforcement arrangements. Put otherwise, DPA needs a 
method of institutional design. I will illustrate this with an example. 

In their article on the public housing system in Vienna, Austria, Wagenaar and 
Wenninger describe how that system not only successfully houses 60 per cent of the 
Viennese population, in the process creating a liveable, mixed-use city, but has done 
so for a century now (Wagenaar and Wenninger 2020). Viennese public housing rests 
on five integrated strategies: (1) active social protection in which the availability of 
high-quality, affordable housing and strict tenant protection laws are a key feature 
(2) a proactive land purchase and zoning policy; (3) a multi-pronged finance pol- 
icy that consists of affordable rents, capped construction, and land purchase costs; 
(4) long amortization periods; and (5) reliable, long-standing partners (predomi- 
nantly housing corporations) who agree to a cap on profits (3.5 per cent of invested 
capital) and an obligation to reinvest capital in the construction of public housing 
(Ludwig 2017). 

The authors trace the origins of the system to the remarkable surge of inte- 
grated institution building by the Social-Democratic administration of Vienna in 
the years following the First World War, the famous ‘Red Vienna’ (Das Rote Wien) 
era (Weishmann 2002). They describe how this administration, faced with a series of 
overwhelming social and political challenges that, in today’s language, surely deserve 
the designation ‘sustainability crises’ (Bartels et al. 2020a, 392),° forged a remark- 
ably comprehensive, ambitious, and robust municipal administration system. The 
Social-Democratic city administration was driven by a combination of idealism and 
exigency. It had an integrated vision of the social, cultural, and political emancipation 
of the working class. In designing and realizing its agenda, it built upon the tradition 
and achievements in energy provision, public transport, and tenant protection of pre- 
vious administrations. In effect, what the officials of Red Vienna were engaged in was 
‘design-in-practice’ (Kimbell 2012, 2015). 

Design-in-practice combines many of the themes and strategies of transformative 
intervention discussed so far: 


[Designers] are seen as using an iterative process that moves from generating 
insights about end users, to idea generation and testing, to implementation. Their 
visual artifacts and prototypes help multidisciplinary teams work together. They 
ask ‘what if?’ questions to imagine future scenarios rather than accepting the way 
things are done now. 

(Kimbell 2011, 287) 


Mediating between policy, evidence, and delivery (Kimbell 2015, 9), design- 
in-practice represents a participatory, practice-based approach to systematic 
institutional change (Kimbell 2012, 142). It presumes that we step back from a static 


° Bartels et al. define sustainability crises as ‘large-scale, dynamic, multi-dimensional systems problems 
that require immanent and systemic resolution to prevent irreversible catastrophe (2020a,393). 
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conception of institutions as a fixed set of rules, norms, and understandings, often 
but not exclusively embodied in organizations. If, instead, we employ a transactional 
perspective of institutions as ‘dynamic, ongoing interactions between concepts, expe- 
rience and situations’ (Ansell 2011, 36), we begin to see how institutions can be 
designed for interconnectedness. Institutions are not context-independent entities 
but instead are embedded in and reflect the experiences of particular communities. 
Such culturally embedded institutions ‘accrete in response to a variety of local situa- 
tions and are evaluated by the test of time’ (Ansell 2011, 37). They are chosen, but not 
designed in the sense of rational design, and although the choice is never compre- 
hensive or synoptic, it is guided by a normative vision. Neither wholly sedimented 
into our social and political environment, nor alterable at will, institutions are part 
of a world of meaning and purpose. In this pragmatist vein, Ansell designates them 
as ‘a skein of practices, values and loyalties that are essential to the very meaning of 
the institutions (Ansell 2011, 37). 


Conclusion 


Without excluding other methods, Deliberative Policy Analysis begins to converge 
on an identifiable and replicable methodological approach: Designed Deliberative 
Platforms, the enhancement of the deliberative capacity of policy systems, and 
institutional design-in-practice.° Design-in-practice is the mother lode of all three 
methods (Kimbell 2012). Design-in-practice does not conceive of designers, or pol- 
icy analysts in our parlance, as the main agent in design, but instead opens the process 
up to officials, citizens, and other stakeholders who, through their practices, are an 
intrinsic part of the problem at hand (Kimbell 2012, 135). It understands design- 
ers/analysts as ‘practitioners being in the world and their relation to other social 
actors including artifacts and other social practices and institutions’ (Kimbell 2011, 
298). This relational, decentred view of design and analysis corresponds with both 
the theory and practice of DPA as a methodology that fits the dynamic complexity, 
essential indeterminacy, and dense interconnectedness of policy systems, and that 
accords with the continuity between policy analysis and policy practice. 

These methods represent an important step forward in the development of DPA. 
That said, some important unresolved issues remain. For example, DPA claims that it 
provides a strategy for dealing with complexity, but its theory of complexity, and its 
implications for policymaking, policy inquiry, and democratic governance, reflects 
the underdeveloped state of the art in this field. Second, DPA describes itself as 
a normative-empirical programme, but its ethics are underspecified. Part of these 
ethics are internal to the programme. These are the well-known normative prin- 
ciples of deliberation as a special kind of communication (reason-giving, attentive 
listening, mutual respect, and non-coercion) as well as the ethics of interpretation. 


é And utopian re-imagination. See footnote 3. 
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Fay describes the latter as striving for enhanced communication and professing 
respect for the life world of the subjects of inquiry (Fay 1975)—to which I add that 
this injunction should extend to all subjects of inquiry, including natural objects 
and future generations. However, DPA also embodies a situated ethics. These are 
the values that emerge in the situation at hand and reflect the challenges that actors 
face, their needs and desires, and the affordances and constraints that frame their 
choice architecture. At present, DPA does not offer a coherent understanding of, and 
approach to, integrating contextualized values into policy inquiry. 

Finally, an endemic risk of policy analysis is that its results are not adopted or 
acknowledged by officials in the policy subsystem (see also Li and He 2016, 241). This 
is because the output of analysis suffers the effects of cognitive dissonance, does not 
accord with the political mandate of these officials, is considered by policy practition- 
ers to be irrelevant to their needs or lacking in authority, or because the input from the 
larger policy subsystem is more urgent than the results of analysis. In other words, 
critical, interpretive analysis is not ‘transformative by default’ (Bartels et al. 2020a, 
397). If it aspires to be more than just an epistemic innovation, DPA needs to position 
itself more clearly as a critical-interventionist approach to democratic enhancement. 
DPA as design-in-practice seeks to enhance the effectiveness of policy analysis by 
prioritizing the establishment of new interactions within the policy subsystem, and 
by involving citizens, politicians, and administrators as co-designers. DPA has suc- 
cessfully invited a rethinking of the role(s) of the policy analyst, as less, or perhaps 
more precisely not just, a producer of quantitative or interpretive, scientific knowl- 
edge, a consultant in policy design (Howlett 2011) or a provider of policy arguments 
(Majone 1992), but more a knowledge broker, change agent, and designer and facil- 
itator of deliberative inquiry, evolutionary learning, and democratic process (Ansell 
2011; Bourgon 2011; Bartels and Wittmayer 2018, 9). Yet, DPA has still to demon- 
strate more convincingly the practical feasibility of these roles in the institutionalized 
landscape of public policymaking. 


Further Reading 
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Action research is a form of research in which knowledge claims are co-generated 
with community members with the goal of taking action to promote social change 
and social analysis (Greenwood and Levin 2006, 5). It is a methodology well-suited 
to deliberative democracy as they share similar commitments, assumptions, and 
underlying values. Both action research and deliberative democracy are grounded 
in principles of inclusion and equality, aimed at the co-generation of knowledge, and 
oriented towards action. While action research may not be the best methodology for 
studies on deliberative democracy, it is a particularly suitable methodology for delib- 
erative democracy. By this we mean that while research on deliberative democracy is 
more likely to be done at a distance, action research, as this chapter illustrates, is part 
of the deliberative democratic process itself. 

Deliberative democracy, according to James Fishkin, ‘is the practical answer to the 
philosophical question: What would the people think should be done if they could 
consider key issues under good conditions for thinking about them?’ (2018, 1). The- 
orizing and experimenting with the creation of these good conditions is the work of 
scholars and practitioners in the deliberative democracy field. Within this field, ‘pub- 
lic engagement’ is the term used to encompass a range of methods for creating these 
good conditions for people to be brought together to address issues of public impor- 
tance (Nabatchi and Amsler 2014). In this sense, action research is a methodology 
for deliberative public engagement. 

Methodologically, action research aligns with critical theory and feminism in its 
commitments to emancipatory knowledge-seeking and approach to research partici- 
pants, who are considered not subjects but rather collaborative agents in the research 
process (see Harding 1987). In action research, research partners have the power 
to shape, interpret, and act on co-generated knowledge. Writing about the inter- 
sections between action research and feminist methodology, Colleen Reid points 
to their ontological and epistemological coherence, as both ‘seek to shift the cen- 
tre from which knowledge is generated’ and ‘share an avowed intent to work for 
social justice and democratization’ (2004, 4). It is also possible to draw similar par- 
allels between action research and deliberative democracy, where the purpose of 
research is also to contribute to democracy. Scholars advocating for action research 
in the field of deliberative democracy include feminists like Marian Barnes, who 
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give activism, passions, and practical politics a central place in deliberative prac- 
tices account of justice (see Barnes 2008). Bringing action research into the field of 
deliberative democracy is also an important step towards enacting Hammond's call 
for deliberative democracy to embody democratic critical theory through an inclu- 
sivity that moves beyond mere participation, and a reflexivity grounded in a keen 
awareness of the structures of power at play in any setting where the researcher or 
facilitator shapes the framework of a deliberative exercise (Hammond 2019; see also 
Curato et al. 2018). 

While there are varieties of action research, in this chapter we draw particu- 
larly on the form of action research described in Davydd Greenwood and Morten 
Levin's Introduction to Action Research: Social Research for Social Change (2006), and 
Ernest Stringer’s Action Research (2007). Greenwood and Levin provide an excel- 
lent discussion of the history and epistemology of the action research methodology. 
Stringer, on the other hand, provides a how-to approach to action research. Both 
ground their discussions of action research in an epistemological framework that 
will resonate with scholars and practitioners of deliberative democracy. Greenwood 
and Levin, for instance, describe the action research process as one of ‘democratic 
inquiry where professional researchers collaborate with participants in the effort 
to seek and enact solutions to problems of major importance to the local people’ 
(2006, 62). They go on to enumerate the following five principles that guide the 
methodology: 


1. Action research is context bound and addresses real-life problems. 

2. Action research treats the diversity of experiences and capacities within the local 
group as an opportunity for the enrichment of the research-action process. 

3. Action research is inquiry where participants and researchers co-generate knowl- 
edge through collaborative communicative processes in which all participants’ 
contributions are taken seriously. 

4. The meanings constructed in the inquiry lead to social action, or these reflections 
on action lead to the construction of new meanings. 

5. The credibility-validity of action research knowledge is measured according to 
whether actions that arise from it solve problems (workability) and increase 
participants’ control over their own situation. 


(Greenwood and Levin 2006, 63) 


Ernest Stringer’s discussion of action research deepens that of Greenwood and Levin 
by emphasizing the social values that underpin it (Stringer 2007). He describes it 
as a process of inquiry that is democratic, equitable, liberating, and life-enhancing. 
Echoing many of Greenwood and Levin's points, Stringer emphasizes the iterative 
nature of this kind of research: 
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AR [action research] works on the assumption that all stakeholders—those 
whose lives are affected by the problem under study—should be engaged in 
the processes of investigation. Stakeholders participate in a process of rigorous 
inquiry, acquiring information (collecting data) and reflecting on that information 
(analyzing) to transform their understanding about the nature of the problem 
under investigation (theorizing). This new set of understandings is then applied 
to plans for resolution of the problem (action), which, in turn, provides the context 
for testing hypotheses derived from group theorizing. 

(Stringer 2007, 10-11) 


Action research, in sum, begins with real-life problems, an understanding of which 
is arrived at through deliberative processes that then lead to new meanings out of 
which stem social actions (Cunningham and Leighninger 2010, 63). More than a 
methodology to simply investigate the processes and impacts of deliberation, action 
research is a method for simultaneously engaging in it, as deliberation is embedded 
in the action research process itself (Cunningham and McKinney 2010). 

Thus, action research is a methodology for deliberative democracy, though the 
data generated through the process can also be useful for gaining insight into the pro- 
cess itself. In this sense, then, one could argue that the action research methodology is 
useful in conducting research both for and on deliberative democracy, but it is impor- 
tant to underscore that the latter is always intertwined with the former. This is best 
understood through an illustration of its use. To that end, we draw from our expe- 
riences designing and implementing the Civic Engagement Action Plan process, an 
initiative of the Advocates Coalition for Development and Environment (ACODE), 
a public policy research and advocacy think tank in Uganda. Through this discus- 
sion, we illustrate the common assumptions and theoretical underpinnings of action 
research and deliberative democracy, the way that action research furthers delibera- 
tive democracy, and the ways that data generated through the action research process 
can be analysed and used. 


Action Research and Deliberative Democracy: The Case 
of the Civic Engagement Action Plan in Uganda 


In this section, we provide an example of how action research is used in both design- 
ing and analysing a deliberative process in Uganda, by focusing on the way that 
Civic Engagement Action Plans (CEAPs) are developed in this country. CEAPs are 
exemplary of both the practice of deliberative democracy and action research. Ini- 
tiated as deliberative processes by governments together with research institutions 
that pursue action research, CEAPs are social accountability tools that enable citizens 
to constructively engage with their elected leaders to hold them accountable and to 
demand effective delivery of public services. They are an outcome of civic engagement 
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meetings that are facilitated by civil society organizations and researchers to share 
service delivery standards with citizens’. 

There are three major stakeholders in the CEAP process: citizens, local government 
officials, and researchers. The citizens, together with their elected leaders, identify 
local service delivery gaps and develop plans for engaging the district council to 
address the gaps. Local government councils respond to citizen demands for better 
service delivery and accountability. The researchers, who are associated with local 
civil society, facilitate the process and act as intermediaries between the citizens and 
the local governments by monitoring local government response to citizen demands. 
The CEAP process includes mobilization of participants, organizing civic engage- 
ment meetings, identifying the most pressing service delivery concerns, agreeing on 
the best strategy for engagement, and finally, monitoring the implementation of the 
strategies and the local government's response. 

The purpose of the CEAPs is to enhance the capacity of government to respond 
to citizens’ demands for better service delivery. Conceptually, CEAPs fall under the 
umbrella of deliberative civic engagement, which includes ‘processes that enable citi- 
zens, civic leaders, and government officials to come together in public spaces where 
they can engage in constructive, informed, and decisive dialogue about important 
public issues’ (Nabatchi et al. 2012, 7). Underneath the deliberative civic engagement 
umbrella, the CEAP process fits squarely within the more specific category of direct 
public engagement in local governance, defined by Nabatchi and Amsler (2014, 65S) 
as ‘in-person and online processes that allow members of the public . . . in a county, 
city, town, village, or municipal authority to personally and actively exercise voice 
such that their ideas, concerns, needs, interests, and values are incorporated into 
governmental decision making’ 

What is particularly noteworthy for the purposes of this chapter is that CEAPs are 
grounded in action research; they are designed to move beyond deliberation to ensur- 
ing action. As such, they are also a social accountability strategy in that they are aimed 
towards improving institutional performance by bolstering both citizen engagement 
and the public responsiveness of local governments (Fox 2014, 7). Consistent with 
the findings of research on essential elements of effective social accountability ini- 
tiatives (Grandvoinnet et al. 2015b; Grandvoinnet et al. 2015a), and in keeping with 
action research and deliberative democratic principles of inclusion, the CEAP pro- 
cess involves citizens, leaders of civil society organizations, and government officials, 
and seeks to create a relationship between these three sets of players that is circular 
and ongoing. 

Jonathan Fox (2014) discusses the importance of ‘sandwich strategies’ that create 
a ‘pro-accountability power shift’ through ‘state-society synergy. He argues that, 


1 The CEAPs were developed in the context of ACODE’s Local Government Councils Scorecard Ini- 
tiative (LGCSCI) and subsequently expanded to ACODE’s Center for Budget and Economic Governance 
(CBEG), and later to all of ACODE'’s programmes and projects. 
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While initial opportunities for change are necessarily context-driven and can be 
created either from society or from the state, the main determinant of a subse- 
quent pro-accountability power shift is whether or not pro-change actors in one 
domain can empower the others—thereby triggering a virtuous circle . . . of mutual 
empowerment. 

(Fox 2014, 32) 


The CEAP process was designed to trigger this virtuous circle of mutual empower- 
ment by involving citizens, civil society organizations, and local government officials 
in a direct public engagement process oriented towards change. Through the CEAP 
process, citizens deepen their understanding of the mandated roles and responsibili- 
ties of their local elected officials, better understand their own rights and responsibil- 
ities as citizens, and gain experience using tools of civic engagement. Because CEAPs 
are facilitated by district researchers affiliated with civil society organizations, these 
individuals and organizations deepen their capacities to be important intermediaries 
between citizens and local government officials. They also play the complementary 
role of helping government deliver on its mandate. In this role, they both amplify 
citizens’ voices and monitor government response to civic action. Government offi- 
cials, too, benefit from the CEAP process since they are able to engage with a more 
informed citizenry and receive demands from citizens in a timely manner and in 
forms that they can use. Notably, since 2016-2017, when the CEAP process was 
implemented in all thirty-five districts where ACODE conducts performance assess- 
ments of district councils using an evidence-based scorecard process, average scores 
for councils in the areas of monitoring service delivery and engagement with the elec- 
torate increased by 70 per cent and almost 30 per cent, respectively (Bainomugisha 
et al. 2020). 

The five steps in the CEAP process are: (1) participant mobilization; (2) issue 
identification; (3) information sharing; (4) action planning; and (5) support and 
monitoring. In keeping with the action research framework, and consistent with 
direct public engagement processes, the facilitation of each step must be deeply par- 
ticipatory, involving community leaders and local government officials representing 
the communities in which the CEAPs will be conducted. Moreover, through the pro- 
cess, knowledge generation and action are tied together not just in a way that applies 
existing knowledge, but in a way wherein the research itself generates new mean- 
ings for all participants, leads to social action, and results in improved delivery of 
public services in the areas of education, health, water and sanitation, roads, and 
agricultural services. All along the way, researchers affiliated with civil society organi- 
zations facilitate the co-generation of knowledge, share relevant data and information 
for deliberation, record the knowledge generated, and monitor government respon- 
siveness. The resulting data are both qualitative (fieldnotes and case studies) and 
quantitative (tracking sheets). These data are analysed so as to understand what is 
and is not working and, in keeping with the iterative nature of action research, to 
inform the ongoing recalibration of the CEAP process. 
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Rather than describing each of the steps in detail, we provide an excerpt from 
one of the case studies that takes us through the process as it unfolded in one of the 
communities. It should be noted that within the rural Ugandan context, inequalities 
along age and gender lines can make it difficult for women and youth to have their 
voice heard and considered. For that reason, during the action-planning step, the 
larger group is divided into smaller groups of women, men, and youth. The following 
example comes from one of the groups of women. 


Nawaikoke is one of the rural sub-counties of Kaliro District in the Eastern part of 
Uganda. The main economic activity of the residents of Nawaikoke sub-county is 
agriculture. Over recent years, the sub-county has continued to suffer from low 
productivity and yield from agricultural practices due to long droughts resulting 
from climate change, land fragmentation, and depleted soil quality. A CEAP ses- 
sion was convened on 11 July 2017 to discuss service delivery issues and the 
roles of both citizen and local government officials in ensuring that services are 
delivered efficiently, effectively, and equitably. During the deliberation, criticism 
of the disbursement of agricultural inputs through Operation Wealth Creation 
(OWC)—Uganda’s agricultural extension system at that time—arose as a particu- 
larly significant issue. Citizens pointed out that women were often left out during 
the distribution of seeds by OWC staff, and that seeds were distributed during off- 
season for planting. The women in the group developed an action plan for writing 
a letter to the district councillor representing Nawaikoke sub-county to bring to his 
attention their discontent with the manner in which seeds were being distributed 
to beneficiaries. As part of their action plan, the group convened another meeting 
on 17 July 2017 to draft the letter. The letter written during this meeting, with the 
attendance list attached, was delivered directly to their elected councillor. Upon 
receipt of the letter, he pledged to forward their issues to the Coordinator of the 
OWC programme and provide the group with necessary feedback. In his capacity 
as the area councillor, he wrote a letter to the office of the Chief Administrative 
Officer (CAO) informing him of the citizen letter he received. The Senior Assistant 
Secretary, on behalf of the CAO’s office, wrote a letter to the office of the Coor- 
dinator of OWC requesting that the Coordinator allocate seedlings as requested 
by the group of women from Nawaikoke. The councillor provided feedback on the 
process to the women’s group, noting that the OWC Coordinator verbally made 
a commitment that he would prioritize this particular women’s group in the next 
distribution cycle of seeds, even though they had not organized themselves as 
an official farmers’ group as required by OWC implementation guidelines. During 
the month of September 2017, the women’s group reported that they had received 
the agriculture inputs as promised. Specifically, they received 130 kilograms of 
maize seeds and 60 kilograms of bean seeds. Moreover, the women’s group has 
now been recognized as an official farmers’ group and will therefore be eligible to 
receive disbursements of inputs in future. 
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In this example, the entire CEAP process unfolded over a roughly two-month 
period, beginning with what we refer to as the CEAP session on 11 July. The 
researcher plays an important role throughout the entire process, beginning with 
mobilization of the CEAP participants, moderating the community meeting, and 
finally working with the local government to follow up on the progress of the action 
plan. In other words, the researcher attached to the civil society organization acts as 
an intermediator between the citizens and the local government council to improve 
service delivery. It is the role of the researcher to ensure that the meeting is not 
dominated by the political leaders by encouraging citizens to share their concerns. 
The initial session is the piece of the CEAP process most closely aligned with pro- 
cesses falling under the deliberative civic engagement umbrella, as it is in this session 
that citizens, community leaders, and government officials come together in a public 
space to engage in dialogue about pressing public issues. The outcome of that session 
is a set of action plans for using one of the tools of civic engagement, in this case a 
group letter, to request a resolution of a salient service delivery issue. In the example 
above, the first step in the action plan developed by the women was to convene a 
meeting with a broader group of women in the area on 17 July to draft the letter. This 
is a meeting that the civil society organization-affiliated researcher also attends and 
supports as needed. Fieldnotes were taken on the initial CEAP session and all sub- 
sequent meetings. Following the submission of the letter on 20 July, the researcher 
periodically conducted monitoring visits at district headquarters to interview key 
government officials about the status of the community request and recorded infor- 
mation gleaned through these monitoring visits in a tracking sheet on a weekly basis. 
This case example incorporates both fieldnotes and tracking data. 

In addition to cases like this where issues were resolved through local government 
responsiveness to community demand, there were also many cases—unexpectedly— 
where the CEAP process led to communities resolving issues on their own. In one 
community, for example, the issue of overgrown roads was identified as a problem 
affecting people’s access to markets and health services. A youth group from the com- 
munity decided to organize a workday and clear the bush from the problem areas. In 
another community, the issue of pupil absenteeism due to illegal child labour prac- 
tices in a nearby quarry came to the fore. A group of women decided to resolve the 
issue by going to the quarry and confronting its operators and threatening to call the 
police if they continued to employ children. The practice halted immediately, and 
the children were back in school the next day. There were also many cases where the 
elected leaders present at the CEAP sessions were able take the issues back to the 
district and get the issues resolved without the need for a formal petition or letter. 
In all these cases, the researcher maintained close contact with the community and 
was therefore able to document the progress even in situations where solutions were 
provided to the citizens in record time. 

This description of the CEAP process illustrates the parallel assumptions between 
action research and deliberative democracy. The process was inclusive, designed to 
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co-generate knowledge and new understandings of community issues, and oriented 
towards action. Moreover, consistent with processes falling under the deliberative 
civic engagement umbrella, the CEAP process created opportunities for citizens, civil 
society leaders, and government officials to engage in constructive and decisive dia- 
logue about public concerns. Through this process, citizens identified service delivery 
issues and used the tools of civic engagement to raise their concerns and hold their 
leaders accountable for addressing them. In many cases, the deliberations themselves 
led to citizens and/or government officials taking action to resolve the identified 
issues without the need for letters and petitions. 


Generation, Analysis, and Use of Action Research Data 


Typical of action research, the data generated through the implementation and track- 
ing of the action plans have been used in a number of ways. Examples include 
advocacy work with local and central government officials, identification and initi- 
ation of promising local governance practices, documentation of outcomes, and the 
iterative understanding and improving of the process itself. 

Understanding and advocating for citizens’ service delivery priorities. An analysis 
of the content of the 385 action plans developed by citizens across the thirty-five 
districts in 2017 led to an understanding of citizens’ service delivery priorities. Access 
to health care was the focus of 108 of the plans; 89 plans focused on water; 76 on 
education; 69 on roads; and 19 on agriculture. Within each of these service delivery 
areas, the data point to issues of specific concern. The top three issues in the health 
sector, for example, were inadequate service provision at the health centres, having 
to walk long distances to the health centres, and inadequate supply of medicines, 
in that order. Within the water sector, the most pressing issues identified for action 
were too few water sources, contamination of water sources, and dysfunctional water 
sources. Armed with this kind of information, the district researchers, and ACODE 
more generally, have pressed decision makers at both the national and district levels 
to prioritize these issues in terms of both resource allocation and action. 

Tracking progress and refining the process. The tracking data generated through the 
support and monitoring phase of the CEAP process are useful in gaining insight 
into the process itself, and in identifying which elements should be maximized and 
which need to be rethought. Tracking data on the 385 action plans, for example, were 
analysed to determine which tools of civic engagement (petitions, letters, meetings, 
SMS messages, etc.) were most likely to lead to government responsiveness. That 
analysis, which showed that petitions were the most effective tool, has led to more 
emphasis being placed on that particular tool. A petition template was developed 
and recent CEAP sessions have included more discussion about what Uganda's Local 
Government Act has to say about how petitions should be submitted, received, and 
addressed in district council proceedings. Equipped with this information, citizens 
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are better able to hold their leaders accountable for responding to their petitions. 
Moreover, collecting the tracking data is itself a form of accountability as the civil 
society organization researchers monitor government response. 

Understanding particularly powerful elements of the process. Data analyses also 
point to elements of the public engagement process that are critical for productive 
outcomes. For instance, analysis of the transcripts of interviews with local govern- 
ment officials as part of the monitoring process, and fieldnotes from CEAP sessions 
and subsequent check-ins with community groups, pointed to the importance of local 
government leaders’ involvement in all phases of the direct public engagement pro- 
cess. By design, the CEAP methodology has local elected leaders engaged from the 
very beginning in mobilizing their constituents for the community meetings. They 
are also expected to attend the CEAP meetings to share information about the district 
development plans, explain their role in overseeing service delivery, and respond to 
citizen queries. Not only does this level of engagement lead to increased buy-in on 
the part of elected leaders, an increased level of citizen trust in their leaders, and an 
increase in confidence on the part of citizens that their engagement is worthwhile, 
data showed that it can also make local government more efficient. An interview 
with the Speaker of Council in one district, for example, noted substantial savings in 
resources allocated for monitoring service delivery because the majority of his coun- 
cillors were attending CEAP meetings where concerns could be raised and feedback 
and updates provided. 

Documenting change and sharing promising practices. Data generated through the 
monitoring process have also shed light on how council deliberations are chang- 
ing as a result of the CEAP. Evidence from thirty-two out of thirty-five district 
council meeting minutes where CEAPs were implemented revealed that most of 
the service delivery issues raised in the CEAPs were discussed during the council 
meetings. Analysis of the data also identified practices that can be shared with other 
districts. In the year before the CEAPs were implemented, data collected through 
ACODE'’ Local Government Councils Scorecard Initiative revealed that only one of 
the thirty participating districts had received a petition from citizens. By the end of 
2018, districts were receiving an average of five petitions per year, and Speakers of 
Council in all of the twenty districts monitored confirmed that citizens’ concerns 
raised in petitions and letters now form a regular part of the agenda for council 
meetings. In Nebbi District in Northern Uganda, the deliberation of citizens’ con- 
cerns in council meetings even led to prioritizing citizens concerns about water 
and education in the Financial Year (FY) 2018/19 budget. After receiving a petition 
from citizens in one sub-county, which suggested funds be allocated towards drilling, 
construction, and rehabilitation of boreholes in thirteen villages, the council allo- 
cated UGX 336 million (US$90,000) for the work. Citizens in another community 
in Nebbi drafted a petition to the district council requesting that the council allo- 
cate UGX 15 million (US$4,000) to support regular inspection of schools as a means 
of curbing rampant teacher absenteeism. In response, the district allocated UGX 
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52.1 million (US$14,000) in the district budget for FY 2018/19 for both inspection 
and monitoring of government-aided schools. The actions by Nebbi District Council 
have been shared with other districts as evidence of how the principles and practices 
of responsiveness can be embedded into the way councils do their business. These 
actions were shared during the sessions when results of the performance scorecard 
were disseminated in each of the thirty-five districts where ACODE’s Local Gov- 
ernment Councils Scorecard Initiative (LGCSCI) is implemented. They were also 
shared with the Uganda Local Government Association, which then disseminated 
the information to the other eighty districts in Uganda where LGCSCI is not in 
operation. 

Understanding and addressing shortcomings of the process. Data also point to the 
shortcomings in the process. For example, in nine of the twenty districts monitored 
in 2018, Speakers of Council had not provided feedback to petitioners upon receiv- 
ing their petitions. This affected the subsequent round of CEAPs when a number 
of community members in these districts claimed that the previous processes had 
not yielded much, and they were no longer eager to engage with their elected lead- 
ers. The frustration was not so much because their service delivery concerns had not 
been addressed, but rather because there was no response at all from their elected 
leaders. In response to these findings, researchers worked with Speakers of Council 
in the nine districts to strengthen the feedback mechanism. The monitoring data also 
revealed that information received from citizens did not make it through appropri- 
ate channels in seven of the twenty districts. In one district, for example, the council 
chair informed the CEAP monitoring team about a decision that had been taken 
by the District Executive Committee regarding resources allocated for the establish- 
ment of youth-friendly corners in all health facilities. In a separate meeting with the 
District Health Officer, he noted that he was not privy to this information, yet he is 
responsible for implementing the decision. When the district researcher followed up 
with the chairperson, he admitted that his office had not interfaced with the technical 
staff, but pledged to make improvements in this area. In both these examples, research 
findings led to improvements in local governance practices and to improvements in 
the CEAP process as areas for additional capacity building were identified. 


Conclusion 


Action research is not just a methodology to study the processes and practices of 
deliberative democracy; the former is a vehicle for the latter. Enabling members 
of the public to exercise their voice, such that their ideas, concerns, and needs are 
incorporated into governmental decision-making, the CEAP process and action 
research are examples of ‘direct public engagement’ in action (Nabatchi and Amsler 
2014). In this methodology, the lines between research and deliberative action are 
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blurred. The research itself is designed to engage participants in the co-generation 
of knowledge that leads to social action (Greenwood and Levin 2006) through a 
research process that is democratic, equitable, and life-enhancing (Stringer 2007). As 
the CEAP example shows, the data generated through the deliberative public engage- 
ment process can be either qualitative or quantitative, and analysed and used in any 
number of ways. The data can provide the basis for formative and summative assess- 
ments of the public engagement process, advocacy work by civil society organizations 
or the communities themselves, holding governments accountable for the provision 
of public services, and the identification of promising practices for replication by 
other actors. 

While action research is an excellent methodology for deliberative democracy, it is 
not so well-suited as a methodology for research on deliberative democracy. That is, it 
is less able to analyse deliberative processes from a distance because action research 
is a methodology that is connected to both the design and implementation of the 
deliberative process itself. Thus, it would not be a good choice for analysing the 
effectiveness of parliamentary debates or public engagement components of local 
government meetings. Rather, as the discussion and case study presented in this 
chapter demonstrate, action research is a methodology that generates data through 
the deliberative process itself. When fused together, the shared commitments to 
inclusion and equity, the co-generation of knowledge, and social change are centred 
and enacted through the research process itself. 
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Interviewing is a popular social research method because it allows researchers 
to understand the subjectivity of participants. However, interviews can also rein- 
force existing power relations between researchers and respondents, especially if the 
respondents come from vulnerable or marginalised communities (Auerswald et al. 
2017). One way to address this is to translate deliberative democracy’s norms into 
a research method that places equality, reason-exchange, and listening in the fore- 
ground. This is the case with so-called deliberative interviews (Berner-Rodoreda et al. 
2018). The Community of Inquiry (Col) introduced in this chapter offers one example 
of this approach, designed with a specific focus on minimizing the risk of unequal 
power relations. Col was originally pioneered by pragmatist philosophers such as 
John Dewey (1938) and Charles Sanders Peirce (1955), as a question- and listening- 
centred deliberation. In Col, participants discuss a common topic by exchanging and 
weighing reasons, as well as by questioning and listening to each other for the pur- 
poses of gaining a deeper understanding of the topic at hand. Participants serve both 
as interviewers and interviewees without the researcher’s active involvement. They 
engage not only in responding to questions but also in the activities of questioning, 
listening, and reflecting. 

For deliberative democratic researchers who use in-depth interviews to under- 
stand and examine the subjective views and lived experiences of deliberators, the 
idea of Col is instructive. Rather than conducting usual one-to-one interviews, the 
researcher applies deliberative norms to a group interview to help interviewees exam- 
ine their views and behaviours in and around deliberative forums in a non-coercive 
and non-oppressive fashion, which is a key source of ethically trustworthy and ana- 
lytically useful data. In my own research on deliberation in schools, for example, I 
used Col to understand students’ deliberative experiences in and beyond the class- 
room by mitigating various power imbalances between students and the researcher 
(Nishiyama 2018, 2019). Based on my own research experience, this chapter shows 
how question- and listening-centred deliberation enabled by Col, coupled with par- 
ticipants’ roles as both facilitators and interviewers, can help to realize the epistemic 
and inclusive aspects of deliberative interviews. To this end, this chapter outlines 
the theory behind Col and offers a step-by-step practical guide for its implemen- 
tation in practice. Importantly, unlike other methods in this book which are used to 
understand deliberation, Col uses deliberation as a method to gain useful data for 
deliberative research. 

Kei Nishiyama, Community of Inquiry. \n: Research Methods in Deliberative Democracy. Edited by Selen A. Ercan, 
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Deliberative Interviews 


Deliberative interviews are a practical application of deliberative norms to an inter- 
view method that emphasizes the significance of equality, reason-exchange, and 
listening during an interview process. This method is theoretically rooted in Svend 
Brinkmann’s idea of epistemic interviews. Brinkmann (2007) divides qualitative 
interview styles into two categories: doxastic and epistemic. Doxastic interviews are 
single interviews in which there is a one-way relationship between interviewers and 
interviewees. The relationship in doxastic interviews is similar to the clerk-customer 
relationship that is based on the assumption that the client is always right. Inter- 
ruptions, expressions of disagreement, critical questions, and stating opinions in 
response to interviewees’ experiences on the part of the interviewer are prohibited 
because doxastic interviews aim to understand the research subject as accurately as 
possible. Examples of this type of interview include narrative interviews, oral histo- 
ries, phenomenological interviews, and ethnographic interviews (Berner-Rodoreda 
et al. 2018).’ 

Epistemic interviews differ from doxastic interviews in their understanding of 
knowledge. While doxastic interviews regard interviewees as knowledge-holders who 
must be listened to and not challenged, epistemic interviews presuppose that knowl- 
edge is constructed through the interaction between interviewers and interviewees. 
Epistemic interviews are sometimes called Socratic interviews. Following a Socratic 
approach, a deliberative interviewer actively asks questions, requires the justifica- 
tion of positions, and identifies assumptions in order to understand the underlying 
values, beliefs, and preferences of the respondents in a dialectical and dialogi- 
cal manner. Thus, epistemic interviewers assume that knowledge is not something 
waiting to be discovered, but a product of interpersonal exchange. Epistemic inter- 
views value the process of critical and reflective examination of interviewees’ lived 
experiences; hence epistemic interviewers play a more active role. Sometimes they 
challenge the interviewees’ opinions and allow interviewees to challenge interview- 
ers. They ask various questions to clarify the consistency of arguments and underlying 
assumptions. 

Epistemic interviews can be practised in different forms, such as active inter- 
views or confrontational interviews.’ Deliberative interviews, a practical application 


1 While Brinkmann (2007) is critical of doxastic interviews, it is fair to say that doxastic interviews 
play a meaningful role in building and sustaining rapport with interviewees, especially when the inter- 
view is conducted in challenging settings. Vulnerable individuals (e.g. disaster victims, refugees) often 
need a space for expressing their feelings and emotions and being heard without interruption, and such 
doxastic interviews enable a researcher to approach the reality of interviewees (Auerswald et al. 2017). 
Kohli’s (2009) argument on the ethics of researching with unaccompanied asylum-seeking children offers 
valuable insight to understand this point. 

> Active interviews are one method in which an interviewer treats interviewees not as a repository 
of knowledge but as a constructor of knowledge. In this interview, an interviewer engages in collabora- 
tive communications to assist an interviewee’s narrative production and reflective examination on his/her 
own experiences (Holstein and Gubrium 1995). Confrontational interviews are one example of an active 
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of deliberative theory to the interview method, offers another form of epistemic 
interview (Berner-Rodoreda et al. 2018; Curato 2012). Deliberative interviews fol- 
low the norms and ideals of deliberative democratic theory. First of all, deliberative 
interviews are inclusive. In the context of a group interview, the interviewer should 
provide participants with an equal opportunity to present and justify their position, 
to reflectively examine their beliefs, values, or preferences, and, most importantly, 
to be heard by others. Interviewees are encouraged to reason together, to ask ques- 
tions to clarify the meaning of what has been said, and, if necessary, to disagree with 
each other, so that they collaboratively reach a clearer and deeper knowledge of their 
lived experience. Without listening, which is the requirement of respecting differ- 
ent opinions and the desire to understand others, deliberative interviews may end 
up as tokenistic conversations rather than offering an opportunity for intersubjective 
understanding. In addition, to make deliberative interviews inclusive, interviewers 
allow interviewees to express their views with various forms of reasoning, includ- 
ing not only verbal expressions (e.g. storytelling) but also non-verbal reasoning (e.g. 
facial expressions, gestures). 

The second important principle of deliberative interviews is that they are oriented 
towards what Curato (2012, 579-580) calls public spiritedness. Deliberative interviews 
do not address private experiences but discuss public issues or, in Curato’s terms, 
‘common human interests: Interviewees may still speak about their private experi- 
ences. Yet, what is important in deliberative interviews is that participants explain 
and justify (1) how their private experiences relate to the common topic, and (2) why 
their stories and narratives are worth considering together in a deliberative commu- 
nity, even if their experience is not directly related to common interests. As such, 
deliberative interviews can be conceptualized as ‘the joint search for a better under- 
standing’ (Berner-Rodoreda et al. 2018, 6). Interviewers and interviewees are asked 
to consider the relationship between their lived experiences and the common topic 
to jointly share and examine the experiences. 

Deliberative interviews do not sit well with positivist approaches because deliber- 
ative interviews assume that knowledge is reflectively constructed through interper- 
sonal interactions, rather than waiting to be discovered through hypothesis testing 
(Berner-Rodoreda et al. 2018). Instead, deliberative interviews correspond with a 
constructivist, or so-called interpretive epistemology. That is, knowledge, the mean- 
ing of actions, and experience emerge within a broader historical, social, and cultural 
context (Schwartz-Shea and Yanow 2012). As with deliberative interviews, interviews 
used in interpretive research value the active and deliberative involvement of the 
interviewer. The interviewer's active involvement provides an opportunity for the 
interviewee to examine the meaning of his or her knowledge and experiences (Kvale 
2007; Schwartz-Shea and Yanow 2012). However, while interpretive interviews have 


interview, but its focus is an agonistic aspect of communication. In confrontational interviews, an inter- 
viewer challenges an interviewee’s views and narratives to critically examine his/her lived experiences from 
multiple angles (Kvale 2007). 


Kei Nishiyama 453 


no specific normative agenda, deliberative interviews follow the specific norms of 
deliberative democracy. Hence, one could say that deliberative interviews offer a 
normative framework for interpretive research (see also Curato 2012). 


Power Imbalances in Deliberative Interviews 


As we have seen, deliberative interviews can offer meaningful contributions to study- 
ing social phenomena. Before we move on to the question of how to implement 
deliberative interviews, it is necessary to consider an ethical question that is at the 
heart of the study of deliberative democracy—namely the question of power. We 
need to ask: how can deliberative interviews mitigate power imbalances between 
interviewers and interviewees? 

Following the normative principles of deliberative democracy, it is important 
to take power into consideration—not only in society at large but also in partic- 
ular research settings. As already indicated, deliberative interviews should be an 
inclusive practice. However, critics have argued that deliberation can reinforce asym- 
metrical power relations in the real world. For example, Sanders (1997) argues that 
deliberation can potentially reproduce or even amplify existing hierarchies because 
deliberation can advantage the communication style of the majority. Also, some peo- 
ple may not receive an equal opportunity or equal hearing in deliberative processes. 
Karpowitz and Mendelberg (2014), for example, show how existing deliberative 
forums shape and reinforce male-centric communication rules and behaviours, and 
how this can have the effect of silencing women’s voices. 

The prevailing inequalities and power imbalances may become even more com- 
plicated with the active involvement of the researcher in the interview process (Kvale 
2006). This could be the case especially in situations where interviews are undertaken 
with marginalized or vulnerable individuals (see Allen 2017; Danaher et al. 2013). 
Consider, for example, interviews with children (Auerswald et al. 2017; Kohli 2009; 
Nishiyama 2018). Deliberative interviewers pose a series of questions to clarify the 
assumptions, values, beliefs or preferences that underlie interviewees’ actions, which 
can make children feel uneasy as they might perceive the interviewer as an inquisi- 
tor (Auerswald et al. 2017; Ebrahim 2010). In this situation, child interviewees often 
use various ‘counterstrategies’ (Nishiyama 2018) to cut off the interview, including 
maintaining their silence, smiling without responding, or using noncommittal utter- 
ances such as ‘hmm: Put simply, children often dislike interview situations in which 
‘children have less control’ (Punch 2002, 328). 

Asymmetrical power relations can also be observed in group deliberative inter- 
views. Children who have a leadership role in their class may dominate in group 
interviews. As Graham et al. (2012, 12-13) indicate, ‘some children would dom- 
inate the interviews and while the researchers attempted to address this, some 
less vocal children would simply agree with statements made by their partners 
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(Yes, me too): These group dynamics may affect the quality of deliberation in deliber- 
ative interviews. This sort of deliberation is problematic precisely because opposing 
and minority voices are not heard at all. As critics of deliberative democracy have 
warned (e.g. Sunstein 2000), such deliberation may risk producing extreme opinions. 

One particularly promising way of taking power imbalances into account and con- 
ducting deliberative interviews is suggested by the idea and practice of Community 
of Inquiry. The next section focuses on the theoretical dimensions of Col with a spe- 
cific focus on its application in the context of deliberative interviews. This is followed 
by an illustration of how Col can be adapted in practice. 


The Concept of Community of Inquiry 


Col was originally suggested as a purely philosophical concept by early pragmatists, 
such as John Dewey, Charles Peirce, and Jane Addams, as a way to challenge the 
transcendental conceptualization of knowledge. Advocates of Col are critical of the 
idea of universal knowledge independent of one’s lived experience, pointing out that 
knowledge is the product of critical and reflective examination of experiences (Dewey 
1938; Peirce 1955). Drawing upon the principle of fallibilism, pragmatist philoso- 
phers argue that knowledge is always open to new interpretation and reflective 
modification, and therefore it is crucial not to search for ‘true’ knowledge but rather 
to share and examine one’s knowledge from multiple angles in the process of com- 
munication (Peirce 1955). Ontologically, Col is grounded in the idea that knowledge 
is socially and discursively constructed, and epistemologically such knowledge can 
be approached and modified only through communicative interactions (Nishiyama 
2018). 

In practice, Col takes the form of group dialogue. In response to a difficult ques- 
tion, a group of individuals create a community where they participate in a process 
of critical examination of the key concepts behind the question by drawing on their 
own experiences, learning with each other by listening to what others say—a pro- 
cess whereby an intersubjective agreement on the resolution of the question emerges 
(see Seixas 1993). The practice of Col is applied to various contexts. Matthew Lip- 
man’s (2003) idea of a Philosophy for Children (P4C), currently practised in over 
sixty countries and regions (see Gregory et al. 2016), is one example of this. Lip- 
man has introduced Col into educational settings, particularly schools, as part of 
education for critical thinking. In these collaborative dialogues, students engage in 
philosophical conceptual analysis, critical examination of their lived experiences, and 
collaborative knowledge-making. 

I have previously suggested that Col can be used as one form of the focus group 
interview method (Nishiyama 2018). As we shall see, however, in contrast with focus 
groups, Col puts more emphasis on the reciprocal aspect of deliberative interac- 
tions. In what follows, I will suggest that deliberative interviews can benefit from Col. 
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I will present four theoretical rationales as to why Col is a good fit for deliberative 
interviews, and how it can help to face the challenge of power asymmetries in 
interview settings. 

First, following the logic of deliberative interviews, Col is a practice of experience 
examination and knowledge generation. As discussed above, the main objective of Col 
is to examine experience from multiple angles through a collective investigation of 
values, beliefs, assumptions, and preferences (Lipman 2003). More specifically, Col 
facilitates participants’ thinking on untested beliefs, stereotypes, and values, to sup- 
port and question each other for the purpose of gaining deeper understanding. It also 
promotes listening to the other side in order to understand multiple perspectives on 
a given topic (Nishiyama 2018). As a result of this collective and cooperative inquiry, 
participants update their perspectives and generate shared knowledge. 

Second, Col is a deliberative practice. To realize the above goal, Col participants 
engage in deliberation that includes exchanging and weighing reasons and listen- 
ing to others. Col is different from conversation because Col participants need to 
explain not only what they think but also how they think and why they hold particu- 
lar opinions. In order to clarify each other’s untested assumptions, participants share 
their experiences using their own vocabulary and listening to each other. In Lipman’s 
(2003, 20) terms, the ideal deliberative process in Col is to ‘listen to one another with 
respect, build one another’s ideas, challenge one another to supply reasons for oth- 
erwise unsupported opinions, assist each other in drawing inferences from what has 
been said, and seek to identify one another’s assumptions. 

Third, Col is deliberation about shared and common concerns (Lipman 2003). Col 
participants talk about shared, or broadly public, concerns rather than simply focus- 
ing on their personal matters. This does not mean that they should not present their 
private experiences. Rather, their personal stories need to contribute to the creation 
ofa common understanding. Sometimes, participants are asked to tell their own story 
as a form of deliberative reason. As a result, Col brings such public spiritedness to 
the foreground of deliberation. 

Fourth and most importantly, to make the interview process more inclusive and 
less hierarchical, Col emphasizes the significance of reciprocal questioning by partic- 
ipants. Early pragmatist philosophers argued that knowledge may not be generated 
or even updated without recognition of different and diverse perspectives (Dewey 
1938). In order to understand what sort of different perspectives participants have, 
Col participants should engage in questioning in a way that helps their inter- 
locutors examine their experiences and thereby help all participants gain a clearer 
understanding of the issue in question (Lipman 2003). 

A unique aspect of Col is also that interviewees, not interviewers, put questions to 
each other in a reciprocal way to gain a clearer understanding of one another's posi- 
tion. This means that, in Col, participants serve as quasi-interviewers. While Col is 
a form of group interview (Nishiyama 2018), this unique feature can further con- 
tribute to making the interview process more inclusive. In focus group interviews, 
interviewers ask a set of questions whereas interviewees talk (see Morgan 1996). 
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On the other hand, asking questions in Col is an interviewee-centric practice. Once 
someone expresses his or her own view, other participants are encouraged to ask 
questions—preferably three or four questions are asked by different participants. 
While the participant-centric Col is useful in mitigating vertical power relations 
between the researcher and other participants, this does not mean that it provides 
a panacea for all sorts of power imbalances. If the researcher takes a hands-off and 
uninvolved approach during the inquiry, this may fail to rectify the horizontal power 
relations amongst the group. The researcher is, therefore, expected to play a crucial 
role as, what Dillard (2013) calls, a moderate facilitator who is a quasi-participant in 
deliberation and whose role is to ask questions aimed at eliciting content and pro- 
viding opportunities to less powerful participants to express their opinions. As such, 
Col, coupled with a researcher/facilitator, can enable participants to share power and 
avoid a situation in which a particular individual dominates the deliberative process. 


A Step-by-Step Guide to Col 


Existing literature offers some promising suggestions about how to undertake delib- 
erative interviews. According to Berner-Rodoreda et al. (2018, 6), deliberative inter- 
views are structured around ideas such as ‘reasoning together, equal speaking time, 
and so forth. This section adds to their argument by providing a more concrete guide 
for Col as deliberative interviews. On the basis of the theory and practice of Col 
(Dewey 1938; Lipman 2003; Peirce 1955) and my own work on Col as an interpretive 
interview method (Nishiyama 2018), I divide the process into six steps: 


Step 1: Preparation 


As with all interviews, the work starts before conducting the interviews (Kvale 2007). 
At the beginning of the group deliberation, all participants, including the researcher, 
sit in a circle so that they can see each other’s faces. To foster a relaxed environment, 
an icebreaker can be introduced at this stage, such as greetings, self-introduction, 
15-second speech, and so forth. As Young (2000) rightly notes, it is recommendable 
to start deliberation with such activities as they enable participants to recognize the 
presence of others. 


Step 2: Instruction about Deliberative Norms 


As indicated by some deliberative scholars (see Gastil and Levine 2005), instruc- 
tions about ‘good deliberation’ offered to participants before deliberation takes 
place can affect the quality of deliberation. Hence, in Col the researcher provides 
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a brief introduction about deliberative norms. These norms include, for example, 
reason-giving, justification of positions, listening, respect for difference. Above all, 
reciprocal questioning needs to be introduced as the core element of Col as a delib- 
erative interview practice. The researcher either explains or demonstrates through 
role play examples of reciprocal questioning that facilitate further collaboration 
and reflection. For example, Kono (2014, 128-129) suggests the following types of 
questions: 


a) Clarification (What is the meaning of X?) 

b) Reason (Why do you think so?) 

c) Evidence (For example?) 

d) Authenticity (Why do you think X is true?) 

e) Generalization (I have a different example about X. What do you think about 
this?) 

f) Clarifying assumption (What makes you think so?) 

g) Inference (If what you have said is true, what will happen then?) 


Step 3: Collection and Selection of Questions 


At this stage, the researcher introduces the topic of deliberation (e.g. students’ self- 
understanding of democracy). The particular question to be discussed, however, is 
selected by participants themselves on their own terms. In the question collection 
phase, each participant is asked to propose at least one question, and the facilitator 
asks ‘why do you propose this question?’ to clarify the participant’s underlying inten- 
tion. The questions introduced by interviewees articulate their particular interests 
in the topic, which provides a meaningful starting point for deepening their collec- 
tive understanding of the topic. Interviewees are asked to pose as many questions 
about the topic as possible (e.g. Is there democracy in our school? Why are we not so 
interested in democracy? What is the meaning of democracy in our lives?) The ques- 
tions generated by the participants should preferably be open-ended (not yes or no 
questions and either/or questions) so that everyone can potentially contribute to the 
deliberation by examining their experience-based knowledge, values, and beliefs. The 
questions introduced by interviewees are written on a whiteboard so that the ques- 
tion remains visually present during deliberation. It is important for all participants, 
including the researcher, to understand the underlying values that give meaning to 
the question. Hence, when each interviewee poses a question, the researcher asks him 
or her to explain why s/he is interested in this question. Then, interviewees choose 
the most adequate and/or interesting question by vote or consensus. The selected 
question serves as the core question of the deliberation. The other generated ques- 
tions serve as sub-questions that can be referred to during deliberation over the core 
question. 
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Step 4: Group Deliberation 


All participants deliberate together. They talk, listen, and think together about the 
selected question through a critical examination of their experiences. Depending on 
the context, deliberation can take anything from thirty minutes to two hours. 

The researcher has a dual role as both the facilitator and guide of the deliberation. 
As a facilitator, the researcher stimulates participants’ question-based interactions. 
For example, the researcher encourages participants to ask questions once one 
speaker has expressed their view; proposes questions to assist in a deep examina- 
tion of expressed experiences; gathers counterexamples; asks for evidence; helps to 
clarify misunderstandings; addresses what is still unexamined; and helps to refocus 
the conversation if participants veer off-topic.’ While facilitating deliberation, the 
researcher also serves as a guide in the deliberation process who ensures the epistemic 
and inclusive quality of deliberation. To realize epistemic deliberation, the researcher 
asks participants to give further reasons to clarify their assumptions and positions, 
and fleshes out incoherence or vagueness in what has been said. To realize inclu- 
siveness, the researcher asks talkative participants to listen and think, provides silent 
participants with the opportunity to express their view, and encourages participants 
to take various forms of reasoning (e.g. storytelling, joking, emotion) into account. 


Step 5: Analysis of the Content of Deliberation 


After the deliberation, the recorded discussions are transcribed. Then the researcher 
analyses the content by employing analytical methods suited for the respective 
research purpose. There are various ways of analysing interpretive data, such as qual- 
itative content analysis, narrative analysis, phenomenological description, discourse 
analysis, frame analysis, amongst others (Ercan et al. 2017; Kvale 2007; Schwartz- 
Shea and Yanow 2012; Wagenaar 2011; see also Boswell, Chapter 23 and Mendonca 
and Simoes, Chapter 24 in this volume). 


Conclusion 


In this chapter, building upon the existing study of deliberative interviews, I offered 
the first step-by-step guide to deliberative interviews with a specific focus on Col. 
I argued that Col enables the deliberative interviews to be truer to the principles of 
deliberative democracy in two important ways. 

First, Col emphasizes the significance of power-sharing through reciprocal 
questioning undertaken by the interviewees themselves. Interviewees ask ques- 
tions in a reciprocal manner to share powers in the deliberative community and 


> Some useful facilitative designs are illustrated in Nishiyama et al. (2020). 
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serve as quasi-interviewers. In doing so, Col reduces the risk of a particular 
individual—including the researcher—dominating the deliberative process. Yet, this 
does not mean that Col leaves deliberation entirely up to the interviewees. Instead, 
the second key insight is that even though Col is an interviewee-centric activity, there 
are various roles that the researcher must play throughout deliberation. Researchers 
need to serve as a guide/facilitator who checks whether deliberation is being practised 
in an inclusive manner and ensures the epistemic quality of deliberation. 

In this way, Col as a deliberative interview method can help deliberative demo- 
cratic researchers to examine and unpack the views and the contextualized meaning 
of the experiences of people in and around deliberative forums. However, it would be 
wise to avoid using Col when the researcher does not have a firm understanding of 
what constitutes authentic and inclusive deliberation. In this sense, facilitation in Col 
is quite a demanding task, and thus Col is not available for everyone. Also, Col is not 
a stand-alone interview method in that it requires the researchers to be familiar with 
the research subject and field under study. In particular, prior to Col, the researcher 
needs to identify who can potentially be (less) powerful during deliberation, and, 
preferably, establish a rapport with the research subject. In this sense, establishing 
the conditions under which Col works effectively requires time and effort, but the 
potential gains to be had in terms of knowledge and data are significant. 
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32 
The Deliberative Camp 


Donatella della Porta and Andrea Felicetti 


The Deliberative Camp is a novel method of inquiry to investigate the complex and 
ever-changing ways in which deliberative democracy and social movement politics 
interact, but also for the promotion of knowledge and activism. This method entails 
the development of an open space for citizens, including activists, and other actors 
that might be interested in coming together to discuss a given political problem 
and envisage future action. Crucially, the Deliberative Camp is inspired by social 
movements own practices, departing from the formalized settings of traditional 
deliberative assemblies. In this sense, the Deliberative Camp can be used as a resource 
for developing and/or investigating grassroots engagement. In this chapter we focus 
on the latter aspect and discuss the Deliberative Camp as a method of inquiry to 
study the role, forms, and qualities of democratic deliberation in social movements. 
Researchers can rely upon this method (in combination with others) to address a set 
of research questions, such as: What are the deliberative qualities of citizens’ inter- 
actions within specific settings? What are democratic deliberation’s strengths and 
weaknesses in the context of social movement politics? Which are the arguments 
used on specific issues and how can they be combined? Which type of knowledge is 
produced during the discursive interactions? 

The Deliberative Camp represents a particularly timely methodological contribu- 
tion, given that deliberative democracy and social movements are increasingly inter- 
connected. The long-standing ties between these two fields have recently resulted 
into substantial cross-fertilization of ideas between social movement studies and 
research on deliberative democracy. In particular, we have seen a growing number 
of theoretical and empirical efforts seeking to ascertain the deliberative democratic 
qualities of movements (e.g. della Porta and Rucht 2013; Felicetti 2016) and activists’ 
contributions to deliberative systems (e.g. Curato 2020; Smith 2016; Parry 2017). 

Today, scholarly developments are more and more accompanied by sustained 
interaction between movements and forms of public deliberation in real life (della 
Porta and Doerr 2018). With particular focus on progressive social movements, 
research has long stressed the role of activists in promoting alternatives to repre- 
sentative democracy and demanding democratic participation and deliberation in 
political institutions. Participatory budgeting is the most well-known and widespread 
example of this (Wampler 2010) and Extinction Rebellion’s call for Citizens’ Assem- 
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blies to fight climate change is one of the most recent illustrations of the role of social 
movement actors in spurring democratic innovation (Westwell and Bunting 2020). 
Furthermore, movements not only express criticism of real, existing democracies and 
support change, but also experiment with different democratic practices within their 
own organizational life (della Porta 2013). Activists’ recent engagement in prefigura- 
tive, innovative, democratic practices, and in forms of deliberative democracy in their 
own activities (e.g. Mendonça and Ercan 2015), reinforces movements’ long-standing 
interest in experimenting with new forms of democratic life (Polletta 2002). 

The Deliberative Camp offers a means to learn from the way democracy is prac- 
tised in real-life settings. This, in turn, can help in developing what Simone Chambers 
labels in Chapter 2 of this volume ‘applied constructive theory, defining it as a way 
of theorizing that gives centrality to the operationalization of ideas from a theory 
and subsequently the development of normative contributions that might advance 
the original theory. In particular, in order to use the idea of deliberative capacity to 
study social movements, we need theoretical refinements, which enable us to observe 
strengths and weaknesses of deliberation in the public sphere. Insight from this type 
of observation helps us recast our ideas on what we ought to (or cannot) expect from 
activists and other public sphere actors in deliberative democracy (Felicetti 2016) or 
even to rethink some essential features of democratic deliberation (Mendonca et al. 
2020). 

Below we offer a quick glance at some key developments that, besides being impor- 
tant in themselves, have greatly contributed to our own thinking about this method, 
from Global Justices social forums of the early 2000s to the recent acampadas of 
the anti-austerity movement. Then, we discuss the basic ideas informing the method 
and illustrate its potential application in the study of bottom-up responses to the 
COVID-19 crisis. Movements teach us that being able to adopt and adapt practices 
and ideas is fundamental for effective democratic engagement. We believe this is an 
important lesson also for those interested in the empirical study of social movements. 
Attentiveness to movements’ ever-changing practices and a deliberative spirit of crit- 
ical reflection might well lead researchers and activists alike to revise and improve 
our proposal. Indeed, as novel practices from movements might offer new means to 
study deliberation in movements, we encourage researchers to envisage the Delibera- 
tive Camp in ways that might, to a greater or lesser extent, depart from the conception 
we present here. 


Social Movements and Research on Democracy 


A quick overview of key developments in research on social movements and democ- 
racy is necessary in order to understand the genesis of the Deliberative Camp. In this 
respect, the labour movement in Europe, which developed during the first phases 
of democratization up to the 1920s, provides a good starting point. Amongst its 
claims, the labour movement articulated a critique of the then dominant conception 
of democracy, as based upon (limited) electoral representation, proposing alternative 
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visions and practices. While struggling for universal enfranchisement, labour move- 
ment organizations also contested the representative model, calling for direct forms 
of democracy and limits upon the delegation of power. As historians such as E. 
P. Thompson (1991) and William Sewell (1980) have observed, the idea of direct 
democracy was kept alive in the labour movement, with frequent contestation of the 
increasing bureaucratization of left-wing political parties and trade unions. Direct 
democracy remained central in the visions and practices of the new social move- 
ments that emerged in the late 1960s and early 1970s, criticizing, amongst others, 
the institutionalization of the Old Left and proposing more participatory democratic 
practices. In promoting their claims, student movements, women’s movements, and 
environmental movements also articulated fundamental critiques of conventional 
politics, thus shifting attention from politics itself to meta-politics (Offe 1985). 

These movements introduced new approaches that departed from the conceptions 
then dominant in the labour movement. In particular, in grassroots social move- 
ment organizations, activists were said to ‘[expect] each other to provide legitimate 
reasons for preferring one option to another. They strove to recognize the merits of 
each other’s reasons for favouring a particular option . . . the goal was not unanimity, 
so much as discourse. But it was a particular kind of discourse, governed by norms 
of openness and mutual respect’ (Polletta 2002, 7). Consistently with the debates 
in political theory, these values and related practices were seen as partial solutions 
to the limits of participatory (especially direct) democracy, such as the ‘tyranny of 
the majority; the foreclosure to newcomers, and ‘hidder leadership (Freeman 1972; 
Breines 1989). 

More recently, the Global Justice Movement has focused on the development of 
intense and innovative forms of networking, the endorsement of diversity and sub- 
jectivity in movements, and the diffusion of multiple repertoires of action, including 
experimentation with forms of ‘possible utopias. In this context, the values of delib- 
erative (or discursive) democracy, with particular emphasis on consensus building, 
have been linked with participatory ones (della Porta 2009). 

Finally, in the anti-austerity movements, protest took a markedly prefigurative 
character. The future social relations that the social movements wanted to construct 
were enacted within protest camps, temporarily occupied spaces that had a cen- 
tral role in mobilization. Those who protested in squares from Tahrir, Kasbah, Sol, 
Syntagma, or Zuccotti criticized representative democracy as corrupted, experiment- 
ing with different democratic models. These were inspired by past participatory and 
deliberative models of mobilization. In part, however, activists built on these models, 
in a process of collective learning and adaptation to new internal and contextual chal- 
lenges (della Porta 2013). In particular, the acampadas—at the same time a repertoire 
of protest and an organizational form—represented a major democratic experiment, 
one that was adopted and adapted in different contexts. 

The acampadas can be seen as an adaptation and development of the social forums 
invented a decade earlier by the Global Justice Movement (della Porta 2009). Acam- 
padas, in particular, added to the focus on participation from below a special concern 
with the creation of inclusive and egalitarian public spheres. The main site of protest 
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was kept in public squares to emphasize openness. The movement also stressed the 
importance of the consensual method by applying it to general assemblies, often 
involving thousands of people, rather than just spokes-councils. The consensual, 
horizontal decision-making process was based on the continuously forming small 
groups that were then integrated in the larger assembly. Deliberation through con- 
sensus was thus intended as an instrument against not only bureaucratization, but 
also the tendency to routinization in assemblies and as a way to build a community 
(e.g. Graeber 2013; Flesher Fominaya 2017). 

The acampada, featuring a participatory model of deliberation, is at the basis of the 
Deliberative Camp idea proposed in this chapter. While participatory and delibera- 
tive democracy can be in tension with each other (Pateman 2012), movements show 
that they can also coexist and support each other in practice. 

Once the value of researching movements and their practices is acknowledged, 
a methodological question emerges: How can we specifically study the democratic 
practices of movements? Democracy in movements and its transformations have 
been analysed through the use of various methodologies. To take the example 
of the project Demos (Democracy in Europe and the Mobilization of Society)— 
coordinated by Donatella della Porta between 2004 and 2008, in order to understand 
deliberative and participatory democracy within social movements—surveys, dis- 
course analysis, and participant observation have been used to examine the different 
ways in which democracy is practised by social movement organizations and their 
activists in the context of the Global Justice Movement (della Porta 2009). Research 
using these various methods has yielded solid insights into how democracy is prac- 
tised in social movements and on their claims about changes that could deepen 
democracy as well as on their capacity to effectively influence it (della Porta 2020). 

In our view, however, the empirical study of the deliberative qualities of social 
movements and their contributions to the political system would be greatly enhanced 
by the use of a method designed for engaging and researching movement practices 
specifically. To this end, we propose the Deliberative Camp as a new, bottom-up, 
deliberative methodology. Importantly, the Deliberative Camp is not only a context 
to learn about deliberation in social movements. A fundamental idea of the method 
is that movements are valuable actors for scholars of democracy to learn from (see 
also Doerr 2018). Below, we first introduce the idea of Deliberative Camps, sketch- 
ing out their main features and basic assumptions. Then, we offer guidance on how 
Deliberative Camps can be set up and used in research that aims to enhance citizens’ 
democratic participation. 


The Deliberative Camp 


Assemblies have been the assumed venue for participatory and deliberative demo- 
cratic innovations—from participatory budgeting to citizens’ parliaments (Elstub 
and Escobar 2019)—which often have positive effects on policymaking and on public 
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debate (Goodin and Dryzek 2006). Nonetheless, scholars have highlighted impor- 
tant limitations in their ability to enhance democratic politics at the systemic level 
(Parkinson and Mansbridge 2012). Despite their potential relevance, political sci- 
entists still pay limited attention to alternative, informal democratic engagement 
practices in public spaces. Evidence showing that emergent forms of democracy in 
social movements effectively remedy some of the shortcomings of assembly based 
forms of participation tends to be neglected (Felicetti and della Porta 2018). Deliber- 
ative Camps challenge this orthodox approach, by taking the emerging practices of 
social movements seriously and adapting them in order to improve civic engagement. 

A Deliberative Camp is a deliberative-participatory process in which participants 
collaboratively identify an issue and address it through inclusive and consensual 
practices of communitarian and direct democracy in public, open-air spaces. The 
Deliberative Camp differs from other methods of studying and fostering deliberation 
in some key respects. First, it retains the goals of both participatory and deliberative 
models, while blending them. In this, it differs from many other forms of engagement 
that researchers have studied, in which participatory and deliberative qualities tend 
to be seen as being in reciprocal tension (Fishkin 2018) and whose analyses rarely 
embody the experiences of participants. Second, Deliberative Camps go beyond 
participatory practices that leave no room for deliberation, and yet they remain a 
genuinely bottom-up form of engagement which adapts the forms of participation 
observed in social movements to broader popular engagement. This is quite unlike 
public deliberation, which tends to engage citizens through adapted formats of what 
occurs (or fails to occur) in institutional deliberation as evoked by the ideas of citizen 
parliaments and citizen juries. 

The Deliberative Camp represents then a methodological innovation in that it 
offers for the first time a method specifically devised to study deliberation in the 
context of social movement research. It seeks to establish a method to research 
informal spaces where activists deliberate, on the basis of a broad conceptualiza- 
tion of democratic deliberation and one that is suited to an interpretivist approach 
to research informal settings (Bevir and Ansari 2012; Ercan et al. 2016; Holdo 2020). 
It is designed with the aim of capturing the deliberative dimension of democratic 
practices. 

The Deliberative Camp is a bottom-up approach to foster and research deliberation 
(unlike the bulk of top-down experiments in deliberative democracy). Interestingly, 
as Floridia (2017) has observed, it was social movement ideas from the 1960s and 
1970s that sowed the participatory and deliberative seeds that would flourish in 
democratic theory starting from the 1980s. The Deliberative Camp builds upon the 
generative capacity of movement practices and ideas, also from a methodological 
standpoint. It does so in three different ways. First, the activities of a Delibera- 
tive Camp are a formalization of the practices that social movements have long 
engaged in and that scholars have studied. In particular, the method is inspired 
by the encampments that have been a main form of organization and action for 
many anti-austerity movements (della Porta and Mattoni 2014; Fominaya and Hayes 
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2019). Second, Deliberative Camps are, by and large, run by the actors themselves 
with very limited engagement on the part of researchers—in a way that resem- 
bles the Community of Inquiry approach discussed in Chapter 31 of this volume. 
Third, the Deliberative Camp is interested in some specific movement practices. In 
particular, it focuses on those through which movements seek to promote values 
that resonate with deliberative ideals, including transparency, equality, inclusiveness, 
consensus, argumentation, orientation to the public good, and preference transfor- 
mation. Importantly, while the method shares features with Action Research (see 
Chapter 30 in this volume), by actively engaging the researched groups, the two 
should not be conflated. If some collaboration between researchers and participants 
is important to generate knowledge, and if there is an emphasis on real-life problems, 
the type of deep engagement with communities, characteristic of Action Research, is 
not envisaged in the Deliberative Camp. With the exception of the act of deliberating, 
there is no primary focus on ‘action (Greenwood and Levin 2006). 

In short, Deliberative Camps seek to achieve transparency through engagement 
in public, open-air spaces. They approach equality through communitarian or direct 
democratic practices. In Deliberative Camps, engagement aims at the inclusion of 
the citizenry at large, and preference transformation in the polity. This is in contrast 
with earlier forum-based democratic engagement, which was aimed at movements 
and activists only and relied upon meetings and interaction models connected to 
associational democracy (della Porta 2005). Deliberative Camps employ consensual 
practices in assemblies open to all, rather than just to spokes-councils and representa- 
tives of social movement organizations. As such, they give space to prefigurative and 
emotional argumentation and the ‘construction of the common, rather than abid- 
ing by rationalistic political debate linked to more or less unquestionable notions 
of ‘common good. Finally, while centred around the practices of social movements, 
Deliberative Camps can also be used widely by other civil society actors. A Demo- 
cratic Camp aims at enhancing the ability of participants to work effectively and 
democratically with their partners, citizens, and political institutions. 

Besides its guiding principles, the Deliberative Camp is markedly deliberative as 
a method in that it places deliberation at the centre of the research. As we will see, 
participants engage in a series of activities that are essentially deliberative in nature. 
This does not mean that participants reject other forms of democracy: in fact, the 
Deliberative Camp could be better understood as a form of deliberative participation 
(see also Fung 2005; Lafont 2019). Likewise, though not central, ideas and actors 
around representative democracy are not altogether alien to the Deliberative Camp. 
For instance, elected representatives might be invited to join and participants might 
use the Deliberative Camp to think about how to address existing, liberal democratic 
institutions. 

The latter point is crucial: the Deliberative Camp is not an exclusively academic 
exercise. The Deliberative Camp aims to observe actors in the context of engagement 
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that has a collective purpose. As we will see, the political goal is not pre-given. The 
aim of the Deliberative Camp, besides the fulfilment of the investigation, is to offer a 
space for interested actors to meet, discuss, and, if possible, coordinate their actions. 
The Deliberative Camp provides a context for movements to engage in democratic 
life and simultaneously for the researcher to study these interactions. 

The experimental approach often used in the study of deliberation is based on the 
idea that it is possible to identify independent and dependent variables, to manipulate 
the former and see how this affects the latter. However, today there is a concrete risk 
for innovators to design (and research) democratic innovations that simulate demo- 
cratic engagement fostering mere activation, rather than empowerment (Hammond 
2020). This is neither empowering nor emancipatory, and thus is in tension with 
deliberative democratic ideals (see Lafont 2019). The Deliberative Camp takes this 
concern seriously and rather than creating a top-down, elite-driven format for partic- 
ipation, it strives to keep as much as possible the bottom-up forms of participation as 
the context to research. In the Deliberative Camp, researchers immerse themselves in 
contexts in which movements concentrate and intensify their deliberative activities. 

Furthermore, the notion of researching citizens as entities, about which it is possi- 
ble to make neutral and generalizable inferences, is replaced by a different under- 
standing of participants in democratic innovations. Rather than being bracketed, 
the specificities of participants in their capacities as activists, members of minor- 
ity groups, interested citizens in terms of, for instance, their values, perspectives, and 
interests are given full attention. 

The Deliberative Camp has been planned in the context of a study in the Tus- 
cany region of Italy. The research will explore how activists, civil society actors, and 
citizens can contribute to address the social, political, and economic effects of the 
COVID-19 pandemic through deliberative participation. The pandemic has gener- 
ated a remarkable degree of mobilization from a diverse set of social movements and 
civil society organizations and identifying the set of actors involved in the Deliber- 
ative Camp represents a crucial challenge for the preparation phase, as we will see. 
Importantly, the Deliberative Camp represents a unique opportunity for interested 
actors to converge in a democratic and deliberative space where, at a distance from 
representative institutions, they can envisage the most pressing problems they face 
and explore ways to address them. 

In the following section, we present a step-by-step guide to Deliberative Camps 
inspired by this ongoing research project. These guidelines can certainly be adapted to 
fit other and potentially widely different research directions. The Deliberative Camp 
can be applied to study how the deliberative participation of these actors relates to 
other contemporary challenges, such as migration and integration, climate-change 
adaptation, and economic recovery. Further, the Deliberative Camp need not be tied 
to the local level, but can be used to study the action of movements that are active at 
other levels (national, local, and transnational). 
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How Deliberative Camps Work 
Step 1: Preparation 


Deliberative Camps are highly visible events open to all interested citizens, espe- 
cially those who might be hesitant to join more formal political arenas. Researchers, 
activists, or other actors in civil society themselves can call for Deliberative Camps 
individually or in collaboration. In either case, snowballing techniques are necessary 
to identify organizations potentially interested in participating and committed to the 
deliberative values informing the interactions in the Deliberative Camp. Researchers 
and participants will select a broad area of interest that will be addressed by partic- 
ipants. As far as funding is concerned, the Deliberative Camp follows similar logics 
(and costs) as a deliberative assembly. Funding from outside research agencies and 
institutions should only be sought with great caution, making sure that funders will 
not be perceived as threatening the movements, as might happen, for instance, with 
private actors with a potential interest in the outcome of the Camp. It is vitally impor- 
tant that the Deliberative Camp is a free space, not controlled or manipulated, for 
instance, by economic or political elites. 


Step 2: The Event 


Once the area of interest and sources of funding have been identified, participants 
can be sought. There is no formal recruitment, thus neither acceptance upon offi- 
cial invitation nor random selection are employed. Rather, the event is made widely 
public across a diverse set of social movement organizations and civil society asso- 
ciations. Avoidance of the selection by lottery of participants, which is characteristic 
of deliberative assemblies, is justified on two levels: a more general one and a more 
specific one. At the general level, we recognize that, although not without its lim- 
itations, the idea of random selection is an important component of institutional 
public deliberation and democracy at large (Abizadeh 2020), but we oppose the ten- 
dency to equate this aspect with deliberation. The fact that sortition has become a 
shorthand for democratic deliberation is problematic, especially so for those actors 
in public spheres that are excluded by the lottery system. In this respect, one might 
think of the Yellow Vests’ critique of the Great National Debate, the public delibera- 
tion process supported by President Macron in France in response to unprecedented 
popular mobilization. In particular, the movements contest exercises in public delib- 
eration that retain formal sortition while perpetuating exclusion, often of activists 
themselves (Ehs and Mokre 2020). Sortition, which can be used in more or less radi- 
cal ways (Sintomer 2018), is in fact a choice that needs to be, first, justified and, then, 
properly conducted. In the specific case of the Deliberative Camps, in a situation of 
scarce resources and in a context in which democratic deliberation deliberately tar- 
gets marginalized publics, sortition seem inappropriate. As oppositional publics are 
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relevant for deliberative democracy (Dryzek 1990), if the use of sortition excludes 
them, it should be reconsidered." 

The Deliberative Camp assembly (or assemblies) is then to be held in a public 
space (e.g. a square) and open to all who show up (after registration, so as to manage 
the event properly). The publicity is, in fact, given fundamental value as it improves 
transparency and respects the principle of equal participation. As for the protest 
acampadas, being in an open public place is intended to give a sense of common 
ownership of the space—it is a space that belongs to all and can be accessed by all, 
embedding the idea of a public sphere as being not only visible but also accessible to 
all (della Porta 2015). 

In another departure from the mini-public logic, the Deliberative Camp does not 
task participants with generating proposals, voting, making policy recommenda- 
tions, but rather seeks to forge ties and, if possible, to envisage common ground upon 
an issue. Linking otherwise disconnected publics and their ideas is valuable for delib- 
erative democracy (see Mansbridge 2005; Fraser 1990). Further, we agree that there is 
value in deliberation even if ‘it ends not in consensus but in a clarification of conflict 
and structuring of disagreement’ (Mansbridge et al. 2010, 68). Towards these ends, 
participants will engage in facilitated discussions. In fact, facilitated interaction (by 
either professional moderators or trained scholars) is fundamental. The deployment 
of Deliberative Camps overall relies essentially on researchers and on their ability to 
acknowledge the desire of participants to own and contribute to the process. 

Once participants have gathered in the open public space, they are divided into 
small groups of 10 to 20 which are guided through deliberation by a group facilitator. 
Moderators will instruct participants about the basic features of the event (e.g. the 
overall rationale of the process, the resources available and their sources, the logistical 
arrangements, etc.) and the basic qualities of the deliberative mode of interaction (e.g. 
respect, justification-giving, reciprocity, openness, attentive listening, etc.). They will 
also facilitate the group discussions. In line with movements’ practices, small groups 
can then convene in general assemblies to share insights from the small groups, to 
discuss matters of special importance, and in case a general decision needs to be 
made. 

The first moment of the actual deliberation is for participants, supported by mod- 
erators, to identify one or two themes of special relevance that resonate with the 
overall theme for debate and the agenda of the Deliberative Camp. Then, arguments 
and ideas are exchanged. On the basis of the latter, participants will have time to build 
coalitions, coordinate efforts, and envisage common goals. 

The sessions of deliberation should last no more than three hours and can be 
repeated during the day or over a range of days. During breaks between moderated 
debates, participants will interact with each other and with passers-by who might be 
interested in the event taking place in the square. The expectations and timelines for 
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the Deliberative Camp should be established at the beginning of the meeting and con- 
firmed or revised at the end of the day. The Deliberative Camp ends when a common 
problem and related plan of action is identified or when the impossibility of such an 
outcome is acknowledged. The outcome does not need to be a decision about how 
to influence policymaking. Participants, for instance, may resolve to take action to 
counter marginalization and social exclusion independently of political institutions. 
In the Tuscan case, three appointments will be available with proper facilities and 
reimbursement for participants coming from within the region. 


Step 3: Researching the Deliberative Camp 


Researching the Deliberative Camp can occur already in the preparatory phase, by 
means of ethnographic and interviewing techniques, to see, amongst others, who 
is (not) interested in joining and why. Of course, it needs to be disclosed from the 
beginning that the research aspects are an integrating dimension of the Deliberative 
Camp. Researchers might also decide to engage in survey research (e.g. pre- and post- 
interviews, to study, for instance, opinion change) or plan focus groups with some of 
the participants later on. 

One of the definitive strengths of this method is that data is generated during 
the Deliberative Camp itself. There is no specific research object prescribed before- 
hand by the method. Researchers are, in fact, fully free to decide which aspects to 
focus on and what means are best within a given context. Research activities can 
be planned, to a great extent, contextually within the event. This enables one to 
develop a design that is responsive to the needs of all parties involved, in contrast 
with traditional mini-publics, where the researcher’s needs might dominate those 
of the participants. Possible areas of focus are the qualities and forms of delibera- 
tion, the types of networks that are established, the emerging forms of inclusion or 
exclusion, the effectiveness towards reaching political objectives (e.g. coordination 
of action, drafting of political proposals), the individuals’ experiences of the interac- 
tion setting, the type and extent of participation, and the nature of the interactions 
observed (if allowed) through the video-recording of the event. Deliberative Camps 
can use a mix methods approach, for instance, by integrating survey research with 
focus groups, interviews of participants, and direct observation. 

In Deliberative Camps, informal practices of social movements are expressed in 
ways that can challenge the empirical research. Especially, the nature of the interac- 
tion is deliberately informal and with limited constraints from the moderators, which 
might make it difficult, for instance, to audio-record and transcribe interactions, 
due to background noise in an open space. Also, being constructed for welcom- 
ing discursive interactions, the Deliberative Camp might be poorly suited to gain an 
understanding of other aspects of the rich repertoire of social movements’ action. 
This limit extends to digital practices that are gaining increasing relevance both 
in social movements and in deliberative democracy (Gastil and Chambers 2020). 
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Another challenge relates to the need to acknowledge the agency of participants in 
giving shape to how the event unfolds and thus to promote openness and collabora- 
tion while simultaneously affirming the centrality of deliberative democratic values 
to the movements’ practice. 

As someone in charge of collaboratively running and investigating the event, the 
researcher can refine the method. Yet, such efforts, which are challenging and fall 
clearly beyond the remit of a single researcher, demand highly experienced and col- 
laborative teams to work. We expect that with practice some of the promising sides 
of this new method might be strengthened and some of its weaknesses and limita- 
tions might be redressed. The ambition here is not to develop the ultimate means 
to study deliberation in public spheres, but instead to add an important instrument 
to this endeavour as the public dimension of democratic deliberation gains renewed 
attention in the field. 


Conclusion 


In this chapter, we introduced the Deliberative Camp as a new methodology to inves- 
tigate the relationship between social movements and deliberative democracy. As the 
deliberation-movement nexus gains currency on the ground as well as in academia, 
it is important to have methods specifically envisaged to investigate this dimension 
of contemporary democratic life. In this direction, we have presented the main ratio- 
nale in the development of this method, the key steps in its use, and its main strengths 
and weaknesses. Importantly, the Deliberative Camp is not just a resource that aims 
at studying deliberation in movements. First and foremost, it is a method based on 
movements’ deliberative practices. For all those interested in democracy, activists 
are not just actors to learn about, but also actors to learn from. The rich tapestry 
of democratic practices in movements is a valuable source of insights for delibera- 
tive democracy and for democratic theory more generally (Felicetti 2021), which can 
help to redress the field’s long-standing preference for looking at politics as it unfolds 
in formal institutions, be they traditional (such as parliaments) or innovative (such 
as mini-publics). That is a key concern for an approach to democracy that is charac- 
teristically interested in democratic systems at large, not just institutions (Chambers 
2012). 
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33 
Mutual Need 


In the Study of Deliberative Democracy, Norms, Practice, 
and Empirical Assessment Depend on One Another 


Jane Mansbridge 


In 1998, James Bohman proclaimed in a classic article that deliberative democracy 
had ‘come of age. The field has now reached early adulthood. It still has the ram- 
bunctious enthusiasm of youth, diving into new projects and new ways of seeing, but 
it combines that energy with the sober responsibility of the early child-rearing years. 
What we do now has consequences. In this critical moment, deliberative democracy 
benefits from an unusually close connection between normative theory, practitioner 
invention, and empirical analysis, reflected in this volume. Empirical scholars need 
normative theory in order to identify measures of ‘success: They also need results 
from practice. Practitioners need both normative and empirical guidance in their 
experimentation and replication. Theorists need the input of practitioners, who are 
responsible for many insights and telling advances. They also need empirical schol- 
ars, who expose gaps and problems in normative theories and whose qualitative and 
quantitative findings—both inductive and hypothesis testing—help refine, expand, 
or revise those theories. 

When analysing the necessary recursivity amongst theorists, practitioners, and 
empirical researchers,’ I focus on both perceived and normative legitimacy. Today 
many democracies face crises of perceived legitimacy, prompted by increasing 
inequality, increasing globalization, and, most fundamentally, increasing interde- 
pendence, which often requires increasing government regulation and therefore 
coercion.” The field of deliberative democracy is therefore concerned not only with 
how to solve problems better, with greater epistemic accuracy and creativity, and not 
only with how to bring more citizens generatively into democratic participation, but 
also, at least implicitly, with how deliberative innovations can strengthen the legiti- 
macy of the many new regulations that more interdependent human beings will need. 
A new OECD publication lists ‘stable or rising levels of public trust in government’ as 


* I take this application of recursivity from theorists in the new field of grounded normative theory (see 
Johnson, Chapter 4 in this volume, and discussion below). 

? For a fuller exposition of the increasing need for government coercion, see Mansbridge (2014, 2022) 
and note 12 below. 
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one goal of deliberative democracy.* Normative legitimacy (what we decide is right 
after consideration) should underlie any rise in levels of perceived legitimacy (public 
trust). Accordingly, we may ask: How can deliberative citizen forums increase both 
the perceived and normative legitimacy of regulations that some people will expe- 
rience negatively and perhaps also as coercive? Finding workable approaches to the 
legitimacy deficits of current democracies requires the combined and recursive forces 
of practitioners, theorists, and empiricists in the three arenas of deliberative theory, 
deliberative citizen forums, and deliberative systems. 


Theory 


The study of deliberative democracy has spawned a closer connection between 
normative theory and empirical research than any other field in political science. 
Normative theory is always somewhat inductive.* Even Kant must have built on what 
he saw in himself. Rawls built not only on what he saw in himself and others but 
also on what he learned in academic forums about the empirical world and others’ 
underlying conceptions of justice. Habermas drew early inspiration from the empir- 
ical world of eighteenth-century coffee shops, and continually analysed the political 
world he experienced in Europe in the late twentieth and early twenty-first centuries.” 
The normative theory in deliberative democracy is especially inductive, partly 
because it takes respectful interaction as its normative foundation,° partly because 
it is inspired by the current proliferation of novel deliberative institutions around 
the world, and partly because the theory itself is new. Innovations in normative the- 
ory tend to be inductive because of the way norms develop. Human beings often act 
in new ways, morally/normatively as well as in other ways, without thinking con- 
sciously about why they are doing so. Then they begin to find words for their actions, 
thoughts, and feelings. The next step, ‘normative theory, analyses those conscious 
and unconscious formulations, asking on what normative and empirical foundations 
they are built, and seeing if, once formulated and taken forward, those normative and 
empirical implications might conflict with other norms or empirical realities.’ 


> OECD (2021, 26), drawing from Nabatachi et al. (2012). 

* Here I go further than Chambers (Chapter 2 in this volume, p. 28), who writes that the normativ- 
ity in normative theory ‘can be, and often is, deeply rooted in empirical reality. For theory that is more 
explicitly inductive (e.g. Mansbridge 1980 and much of the theory in the field of deliberative democracy), 
Landemore (2020) has coined the term ‘inductive theory. 

5 See, for example, Rawls’s theory of reflective equilibrium and the many footnotes in A Theory of Justice 
(1971) that respond to his many interlocutors; and Habermas (1989 [1962]), in which he first spells out his 
vision of deliberation in the public sphere, and 1996 [1992] for, e.g., his ‘two-track’ theory and reanalysis 
of bargaining. On the empirical and normative, see also Bowman 2022. 

® See Smith in the Foreword to this volume, on mutual respect amongst fields in the deliberative com- 
munity; and Ercan, Asenbaum, Curato, and Mendonga, Chapter 1 in this volume, especially p. 6 and 
references to Hammond (2019), Lee (2011), Renwick et al. (2017), and Farrell et al. (2019). 

7 See especially Chambers (Chapter 2 in this volume) on ‘critical reflective theory, exemplified in the 
work of Iris Marion Young. 
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Theorists can become practitioners and empiricists in the field of deliberative 
democracy without relinquishing their theoretical work. James Fishkin began his 
career writing relatively abstract normative theory and drawing primarily on the rich 
past literature in political theory. Michael Neblo ‘cross-trained’ in normative the- 
ory and empirical work, writing his first book in normative theory. Both have since 
become active in promoting and even creating new deliberative institutions, which 
then prompt these scholars’ own empirical studies and new theories. 

Practitioners become theorists as they collectively and inductively develop princi- 
ples to guide the practice of deliberation. Recognizing the potential problems in the 
proliferation of randomized citizen forums without normative guidance, practition- 
ers in the OECD, after extensive consultation with other practitioners and theorists 
and immersion in some of the relevant normative theory, have drawn up eleven prin- 
ciples for ‘good practice.* Enunciating these principles is a major step in the world of 
practice. It also advances normative theory. Next steps in this recursive process might 
investigate the implications of, and possible trade-offs amongst, these principles. 

Empiricists can become normative theorists in this field without relinquishing their 
empirical work. The major task of those trained in empirical political science is to 
test theory, and this is the goal of most chapters in this volume, but several empir- 
ical scholars in this volume also contribute explicitly to theory. Empirical political 
scholars help develop normative theory when they use their experiences to think 
hard about both the goal (what we want to achieve; the dependent variables) and the 
potential causes (what produces or hinders what we want to achieve; the indepen- 
dent variables). Quantitative empiricists and modellers are in good positions to help 
refine, expand, or revise the theory, because they have to operationalize the question 
of what we want to achieve, and that process of operationalization prompts analytic 
thought (Chambers, Chapter 2 in this volume). Qualitative/interpretive empiricists 
are also in good positions to refine, expand, or revise deliberative democratic norma- 
tive theory, because, on the question of what we want to achieve, their job is to notice 
what people value and ask why. On the question of causes, they also have in-depth, 
sometimes lived, experience with some of the most subtle threads in the fabric of 
deliberative interaction. 

As Genevieve Johnson points out in Grounded Normative Theory (Chapter 4 
in this volume, p. 52), ‘it is now standard practice for contemporary political 
theorists. . .to engage directly in the development, assessment, and analysis of forms 
of empirical evidence or empirical research materials.’ The goal of ‘recursivity’ 
in grounded normative theory has the theorist ‘moving back and forth’ between 
empirical materials and normative claims, sometimes in dialogue with research part- 
ners and participants, in order to develop, revise, refine and sometimes co-produce 


ê Examples of these principles are (1) a clear purpose, (2) influence on public decisions, (3) trans- 
parency, and (4) representativeness (OECD 2020, 118-119). See also the five ‘Principles of Deliberation 
from the Australian non-profit MosaicLab (2021) and, in a related, overlapping field, the seven ‘Core 
Principles for Public Engagement’ from the National Coalition for Dialogue and Deliberation (2009). 

? See Ackerly et al. (2021), discussed in Johnson, Chapter 4 in this volume. 
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the evolving norms (p. 55; see also Asenbaum, Chapter 5 in this volume). At the 
same time, empiricists in the field of deliberative democracy must engage directly 
with normative theory because, as noted, that theory provides the very basis for their 
explorations. Their empirical work then provides the basis for recursivity in helping 
to develop, revise, and refine the theory. Such recursivity applies to both dependent 
and independent variables in deliberation. 

In asking, ‘What do we want to achieve?’ (and formulating their dependent vari- 
ables), researchers differ—some focusing on process, some on outcome. Historically, 
both deliberative theorists and empirical researchers began with a focus on achiev- 
ing high quality in the process of deliberation. The Discourse Quality Index (DQI), 
discussed in Bachtiger et al. (Chapter 6 in this volume) and explicitly derived from 
the theory of Jiirgen Habermas, measures how frequently one finds in a deliberative 
process qualities that are arguably good in themselves even if they produce negative 
outcomes. Participatory equality is good in itself, deriving directly from the equal- 
ity of respect owed each human being. A high level of justification is good in itself, 
mostly because reason is good in itself and perhaps also partly because thoughtful 
justification is a form of respect for others. Even storytelling may be good in itself 
as part of a good faith effort to communicate to others the reasons for one’s conclu- 
sions. Aiming at a common good is also good in itself, as is aiming at good for the 
least advantaged. Respect for others, a core principle, is good in itself. Constructive 
politics leading to consensus is good in itself as part of aiming at the common good. 
In expanding the DQI, the suggested new values are also arguably good in them- 
selves: equality of consideration and interactivity/responsiveness as forms of equal 
respect, and substantive quality and sincerity as forms of reason and respect."® 

Some deliberative theorists, however, value deliberation not because it incorpo- 
rates forms of human interaction that are good in themselves, but because it produces 
good outcomes—good epistemic outcomes (Estlund 2008; Landemore 2012, 2020), 
outcomes good for the development of the participants, or good effects on the delib- 
erative system, including increased legitimacy for collective decisions. We like to 
assume that processes based on interactions that are good in themselves will also 
produce high-quality outcomes, but analytically we must accept the possibility of 
non-congruence. The standards for the quality of outcomes are independent of 
the standards for the quality of the process (Bohman 1998), and normative the- 
ory cannot decide definitively between good processes and good outcomes. Both 


1° For a table of ‘First generation’ and ‘Second generation process standards for good deliberation, see 
Bachtiger et al. (2018, 4). Other important process assessments include the Deliberative Reason Index 
(DRI), a measure of intersubjective consistency (Niemeyer and Veri, Chapter 7 in this volume) and the 
lexical Listening Quality Index, the last step in which is “The speaker reports being satisfied with the sincer- 
ity of the listener's listening’ (Scudder, Chapter 8 in this volume). This last step comes close to measuring 
the subjective feeling of being heard, an experience particularly important to members of marginalized 
groups (Bruneau and Saxe 2012). Because of its importance to the marginalized, it would be useful if 
future evaluations could ask directly about being heard, as in, e.g., ‘When I spoke today/in this forum I 
felt that I was being heard and my opinions considered; or simply, in the wording of Roos et al. 2021, ‘In 
this conversation, I felt heard by the other(s)’ 
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are important. Some theorists have even proposed standards that incorporate both, 
such as “meta-consensus,’ or agreement on the kinds of reasons and choices involved 
in the decision, and “intersubjective rationality,’ or agreement within sides on the 
reasons relevant to their arguments (Niermeyer and Dryzek 2007). 

Finally, theorists differ regarding the most distant goal, the deepest dependent 
variable, what we ultimately want to achieve. For many deliberative theorists, that 
goal is to promote the emancipation of both individuals and communities from 
domination (Hammond 2019). For many others, traditionally, the goal of good delib- 
eration has been democratic legitimacy (Manin 1987; Cohen 1989).™ To those goals, 
and compatible with them (possibly even another way of phrasing the legitimacy 
goal), I add meeting human needs. In an era of increasing interdependence, meeting 
human needs requires increasing the perceived legitimacy of government coercion, 
and backing that perceived legitimacy with defensible normative legitimacy.'* Good 
deliberation in forums and in systems is a means to each of these ultimate ends: eman- 
cipation, legitimacy itself, and meeting human needs through perceived legitimacy 
backed by normative legitimacy. 

In asking, ‘What produces or hinders what we want to achieve?’ (and formulat- 
ing independent variables), we enter a world of plenty. We can analyse variation in 
existing conditions in the world, variation produced by the plethora of deliberative 
forums now springing up globally, and variation intentionally produced by practi- 
tioners and researchers, in laboratory experiments (Grönlund and Herne, Chapter 11 
in this volume), field experiments (Kingzette and Neblo, Chapter 12 in this vol- 
ume), and innovative forums such as the Deliberative Camp (della Porta and Felicetti, 
Chapter 32 in this volume). The act of choosing and creating these variations, then 
studying them, relies on previous normative thought about what we want to achieve, 
in process and in outcome. It then helps create new normative thought. 

Both good process and good outcomes in deliberation are aspirational ideals, 
meaning that we can rarely or never achieve them fully in practice.’* Kant’s ‘ought 
implies car’ does not apply to reaching these ideals. Because we cannot reach the full 
ideal, we have no duty to reach it. We have a duty to strive towards that ideal; this we 


1 For a detailed and nuanced analysis of these issues, see Bachtiger and Parkinson (2019). 

12 The brief logic (see Mansbridge 2014, 2022) is that increasing interdependence creates needs for 
more ‘free use goods, that is, goods that, once produced, anyone can use without paying (for example, 
toll-free roads, law and order, the common defence, clean air and water, a stable climate; ‘free-use goods’ 
is an umbrella term for what economists call ‘common pool goods’ and ‘public goods’; ‘non-excludable’ 
is technically incorrect). The free availability of such goods tempts many to use them and not pay, creat- 
ing ‘free-rider’/collective action problems that result in the goods being under-produced relative to need. 
When the scale is larger than one in which local and informal sanctions, in addition to duty and solidarity, 
can induce the beneficiaries to pay, getting high percentages to contribute to producing the good usually 
requires some government coercion. The provision of free-use goods is thus the main material reason for 
the existence ofa state, and o ur increasing need for free-use goods entails increasing government coercion. 
Effectiveness requires that the coerced perceive the force that coerces them as legitimate. That perception 
is morally wrong if not backed by good normative reasons. Although a goal of ending domination might 
implicitly suggest that governments, along with oppressive social and economic orders, are the enemy, 
the goal of meeting human needs sees governments as a required means, with governmental and other 
powerful actors as, in the right circumstances, allies. 

1 Kant’s ‘regulative’ ideals; see Mansbridge et al. (2010, note 5). 
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can do.’* Accordingly, as we measure and assess deliberative democratic quality, we 
should realize that failure to achieve the top score, or even perhaps more than just 
above the lowest score, can be normatively acceptable in some contexts. When we 
fail to recognize our ideals as aspirational, promulgating those ideals stringently can 
lower perceived legitimacy. 

Recognizing that our ideals are aspirational is particularly important when we face 
normative trade-offs. On the most operational level, for example, the larger the size of 
a deliberative forum, the easier it is to generate representative accuracy, including the 
representation of greater heterogeneity within subgroups. Large size also improves 
greatly the likelihood of achieving believable results regarding opinion change (on 
statistical significance, see Werner and Muradova, Chapter 13 in this volume and 
Gastil Chapter 14 in this volume). Finally, large size increases the perceived legiti- 
macy of a forum among citizens (Goldberg and Bachtiger 2022; see also Goldberg 
2021). Yet, in addition to its considerably greater expense, which must be drawn 
from other projects that might help the needy directly, larger size impedes participant 
input into the process. The OECD guidelines on evaluations of deliberative forums 
(2021), for example, recommend that participants in these forums have input into 
shaping the process and calling up experts. Such input is far easier in smaller groups, 
especially if the participants have a relationship to the organizers that lasts more than 
one weekend.”* In a separate trade-off, longer time in deliberation often produces 
better process and better outcomes. Yet the longer the time participants must take 
from their work and families, even if they are paid, the less representative a sam- 
ple is likely to become. In Shanghai and Bogota, even relatively short deliberative 
forums with paid participation have had trouble adequately representing the business 
classes,’® and when the pay is low or non-existent, even short forums lose the working 
class and the poor.” Good organizers dig deeper into the lottery to create a stratified 
sample that reflects attitudinal, ethnic, class, and other forms of diversity relevant 
to the issue under discussion, but even then, those who self-select to attend and stay 
through the entire deliberation do not usually fully represent, in other respects, those 
whose outward characteristics they share. The longer the deliberation, the stronger 
such selection biases become. 

The trade-offs that deliberative forums face derive from plural democratic val- 
ues and corresponding plural sources of normative and perceived legitimacy. Good 
deliberative process is one source of legitimacy. So is good output. So is, in the 


14 In Mansbridge (2020), I thank David Estlund for this point. See also Chambers, Chapter 2 in this 
volume. It is not only because of ‘power, interest, and money’ that ‘the real world will always fall short 
even of our mediated realistic ideals’ (Chambers, Chapter 2 in this volume, p. 35), but also because of 
plural and competing ideals. 

15 For this reason, amongst others, multiple assemblies of different sizes may be useful (Bouricius 2013). 

16 For Shanghai, Baogang He, personal communication; for Bogota, Felipe Rey, personal communica- 
tion. 

17 With both government and philanthropic funding, participant pay and the other expenses of delib- 
erative forums must be taken from other good uses of the money, such as funding projects that directly 
benefit the poor. 
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non-deliberative realm, the aggregate process of equal and accurate voting. Trade-offs 
amongst these and other values emerge both in specific deliberative forums and in 
the larger deliberative system. 


Forums 


Deliberative forums of randomly-selected/stratified citizens’? are now the main focus 
of research on deliberation. The recent explosion of innovation in such forums in 
Europe and across the globe has emerged in felicitous combination with global 
gatherings (often via internet) of normative theorists, empirical scholars, and practi- 
tioners aiming to process together the lessons from the trial and error of these many 
experiments and make those lessons available to future practitioners and the world. 
Some rough version of normative theory always precedes experiments in practice, 
but that theory may be no more than an intuitive and possibly incorrect conviction 
that the closer the citizen forum to power, the better.’? Innovations in practice have 
generated and will generate more developed innovation in normative theory. 
Consider some recent innovations in structure. In 2017, organizers designed the 
Citizens’ Assembly on Brexit so that the random sample of fifty citizens from the 
UK spent one weekend learning from information brochures, experts, and the other 
participating citizens, then went home for three weeks before gathering for a second 
weekend of deliberation that ended with votes on several recommendations. The rel- 
atively small size of the assembly made it possible for several participants to approach 
the organizers at the end of the informational weekend to ask if it was correct, as one 
of the experts had reported, that EU rules allow deportation of non-citizens if they 
had not found a job in three months. On hearing that this was correct, the participants 
asked if a option featuring this rule could be included in the set of choices offered to 
the participants for a vote at the next weekend. The organizers agreed, phrasing the 
option as ‘Free movement within the EU with the addition of these controls? After 
the weekend of deliberation, twenty-six of the fifty participants, chosen for attitudi- 
nal as well as demographic representation of the population of the UK, supported 
that option.” The weekend-break-weekend structure had encouraged recursivity 
between participants and practitioners. Normatively, this experience suggests adding 


18 Henceforth the word ‘random will mean ‘near random and stratified; except where otherwise noted. 

1° See Arnstein 1969. Lafont (2019), however, argues normatively against giving randomly selected citi- 
zen forums full decision-making power, a position congruent with that of many citizens (Jaquet et al. 2020; 
Devillers et al. 2021; Goldberg and Bachtiger 2022). I suggest elsewhere (Mansbridge 1980 and subsequent 
work) that normative goals are often contingent on context. 

°° The CAB Report is relatively silent regarding the content of these controls, saying with no further 
detail, ‘Single Market rules do not confer an unconditional right on all EU citizens to reside in the UK, but 
... the UK makes little attempt to remove those who do not have a right to remain’ (Renwick et al. 2017, 
57). This sentence and later broad references to ‘exercising the available controls’ almost certainly refer to 
the specific EU rule stating that visitors from other countries have no right to remain in a host country 
after three months if they have no job and do not have ‘sufficient resources not to become a burden on the 
Host Member State’s social assistance system (European Parliament 2019, 2). 
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generativity—producing ideas that were not previously on the table—to the process 
criteria for good deliberation. 

In another structural innovation, organizers in Bogota designed an ‘Itinerant Citi- 
zens Assembly, meaning that one citizens’ assembly commissioned by the legislature 
sets the agenda, the next deliberates on and decides the policy, and a third evaluates 
the results of that policy, creating recursivity amongst the participants that may prove 
generative and theory-inducing, as well as providing protection from external domi- 
nation. A future innovation in Bogota may introduce a citizens’ assembly composed 
only of women as a form of enclave deliberation. That practice would derive from a 
normative theory regarding enclaves that was itself inductively derived from practice 
(see Mansbridge 1994 and below on enclaves in the deliberative system). 

Many innovations have arisen on another structural feature - the integration of 
citizens’ assemblies with legislative decision-making - without much input from nor- 
mative theory. The relevant normative tension arises clearly from practice. On the 
one hand, incorporating elected representatives in the citizens’ proceedings risks 
both domination and the injection of adversary politics. On the other hand, insti- 
tutional connections with a legislature (or administrators) may have deliberative 
benefits as well as costs; these connections also greatly improve the chances that 
the recommendations of a citizens’ assembly will become law; and the connections 
increase the perceived legitimacy of the assembly in the eyes of many citizens (Gold- 
berg and Bachtiger 2022). Hélene Landemore, a theorist whose central normative 
ideal in this context is citizens ‘representing and being represented in turn’ (2020, 
xvii), favourably contrasts Iceland, which energetically kept legislators fully separate 
from the citizen forums, with Ireland, which pioneered greater integration. No the- 
orist or empirical scholar has, as yet, assessed on deliberative criteria the now great 
variety of relationships between legislative representatives chosen by election and 
citizen representatives chosen through random selection. 

Practice is thus leaping ahead of theory as legislatures experiment with different 
degrees of legislative integration. In East Belgium (the German-speaking Com- 
munity of Belgium, population 77,000), the parliament has created a permanent 
Citizens’ Council of twenty-four citizens, randomly drawn from former Citizens’ 
Assembly participants and serving on a rotating basis for eighteen months, which 
selects topics for deliberation by randomly chosen Citizens’ Assemblies one to three 
times a year. The Citizens’ Council meets afterwards with members of parliament 
to discuss the Citizens’ Assemblies’ proposals, then monitors how well the par- 
liament implements the assemblies’ recommendations. Parliament must provide a 
public justification if its members decide not to implement an assembly recom- 
mendation (Niessen and Reuchamps 2022). This model legally mandates integration 
between two relatively independent bodies, the legislature and the citizens’ assembly. 
In the Brussels Region, the Francophone Parliament has created mixed legisla- 
tive committees, each composed of fifteen members of parliament and forty-five 
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randomly selected citizens.” This model legally mandates integration at the level of 
the committee. 

Other nations are experimenting with connections between randomly selected cit- 
izens forums and individual members of the legislature. In Germany, legislators from 
the 299 Single Member Districts in the country are allowed to host Constituency 
Councils of thirty or more randomly selected constituents meeting for one day to 
deliberate on a current political issue and develop solutions.’” This model legally 
sponsors integration at the level of the district. In the US, members of Congress may 
meet virtually with a randomly drawn sample of 175 constituents for one hour of 
discussion on a single topic. This model encourages integration at the level of the 
district, now financially supported by public-spirited NGOs. Although this last form 
of citizen-representative recursive deliberation lasts only one hour, repeated twice 
a week it has the potential to reach every US citizen in the course of a lifetime.” 
Only this last process has received extensive empirical and normative study; the other 
varieties of integration are still too new to have been studied carefully. Practition- 
ers, theorists, and empirical researchers will need to work together to compare these 
different levels of integration for their effects on epistemically well informed pol- 
icy, generativity, citizen development, and the perceived legitimacy of government 
decisions. 

Other innovations in practice tackle the problem of representativeness. It is by now 
fairly well accepted normatively that citizens’ assemblies should be representative 
attitudinally on the relevant issues as well as on the relevant demographics. Pro- 
ducing such representation in practice is harder (Kingzette and Neblo, Chapter 12 
in this volume). Phone recruitment in many countries is no longer viable because 
telephone scams and other intrusions have reduced response rates on surveys to the 
single digits, at least in the US (Gastil, Chapter 14 in this volume) Efforts to improve 
representativeness include going from house to house with personalized invitations, 
as in the German Constituency Council model, or establishing paid panels of a rep- 
resentative sample of respondents, on the models of the National Opinion Research 
Center (NORC) at the University of Chicago, which uses these panels for Delib- 
erative Polls, and TESS at the University of Pennsylvania (Werner and Muradova, 


*1 See https://www.newdemocracy.com.au/wp-content/uploads/2021/04/RD-Note-Brussels- 
Deliberative-Committees-Model.pdf. I thank David Schecter of Democracy R&D for this reference. 

>? I thank Katharina Liesenberg of Es Geht LOS! (private communication) for these details. 

23 Even in a polity the size of the US, if each elected representative at the federal level met in this way 
with constituents two hours a week, fifty-two weeks a year, in six years they could have had deliberative 
discussions with one-quarter of their constituents. Further analyses will show if the elected represen- 
tatives change their own opinions or actions, perhaps improving the quality of their decisions. If the 
representatives do not change on the basis of what they learn, the normative arguments against sup- 
porting incumbents may carry more weight than the normative gains for citizens. Currently, these 
samples are not stratified but largely reflect the underlying population. See Neblo et al. (2018); https:// 
connectingtocongress.org/; and for the calculation, Michael Neblo personal communication. The Cen- 
tre for Deliberative Democracy and Global Governance at the University of Canberra has replicated this 
practice with two Australian MPs (https://connect2parliament.com/townhalls). 
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Chapter 13 in this volume, p. 196, note 3. Pay for participation is also now relatively 
standard normatively, in addition to reimbursements for travel, housing, meals, and 
childcare. Prevailing norms also mandate interpreters for participants who speak lan- 
guages other than the one used in the forum, and empirical work in another context 
shows that these interpreters could also act as advocates for the marginalized (Doerr 
2018). 

Researchers are now becoming more sophisticated about representativeness. A 
study of citizens who had decided not to accept an invitation to participate in one 
forum found amongst their frequent reasons not only competing engagements and 
desires to spend time on work and family but also scepticism regarding the forum's 
impact. These factors, along with the less frequent reasons of self-diagnosed lack of 
competence on the topic, dislike or fear of public situations and others’ judgements 
of self, and dismissal of the event as ‘politics and ‘manipulation (Jaquet 2017), are 
likely to decrease the representation of disaffected and perhaps other marginalized 
citizens.”* With representativeness so important normatively, empirical work should 
continue to investigate not only how to achieve it but also the effects of better repre- 
sentativeness both on process and on the perceived legitimacy of the forum amongst 
the public. 

Empirical and normative work on representativeness in deliberative forums will 
undoubtedly prompt further theorizing on the related problem of representation, 
or the normatively appropriate relation between the citizens in deliberative forums 
and the public. This issue has provoked some controversy. Landemore (2020), for 
example, claims that the citizens who are randomly chosen for a deliberative forum 
and decide to attend represent the other citizens in the polity as well as or better 
than the elected legislators. Lafont (2019), by contrast, sees such forums as incapable 
of providing the ‘authorship’ that democratic citizenship requires; many citizens 
currently agree (Goldberg and Bachtiger 2022). 

The ‘mirror’ form of descriptive representation that most forums provide is also 
relatively inert vis-a-vis the public. It does not involve actively reaching out to con- 
stituencies, as many elected representatives now do. It lies far from the ideal of 
‘recursive’ representation in which, aspirationally, the representative is in ongo- 
ing, mutually responsive communication with constituents (Mansbridge 2022). In 
most deliberative citizens’ forums, the citizens are engaged for too short a time 
to communicate at all with the public at large or with any subgroup who might 
be considered their ‘constituents. They also did not sign on for this task and 
may not be good at it. Yet representation without constituent communication is 
normatively thin. 


*4 Because one bias now is that these forums attract the more civic-minded, the effects of greater rep- 
resentativeness on the quality of internal deliberative processes might not be all positive. E.g., in one 
forum efforts to include extreme climate sceptics resulted in two of the three sceptics walking out (Hobson 
and Niemeyer 2013). The current biases in descriptive representation within randomly selected citizens’ 
forums are of course many orders of magnitude smaller than those in assemblies of elected representa- 
tives, but elected representatives derive their legitimacy primarily from their direct authorization by, and 
accountability to, those who voted for them, not from their descriptive similarity to the population. 
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Can deliberative citizen forums provide sufficiently good representation to make 
the coercion that results from their decisions more legitimate than it is today? At least 
so far, because the world of practice is innovating so rapidly, this question cannot be 
answered definitively either normatively or by empirical scholarship on perceived 
legitimacy. We can say, however, that whether or not these forums have the authority 
to make decisions themselves, their contribution to legitimacy rests in large part on 
their connection to the larger deliberative system. 


Systems 


In contrast to deliberative forums, many researchers in this volume and elsewhere 
agree that the quality of deliberative systems is hard to assess. In introducing the 
concept of a ‘deliberative system’ in 1999, I did not try to operationalize its key vari- 
ables for the purpose of empirical assessment. I focused on normative assessment 
and argued that although one should judge the deliberative components of a system, 
including forums and everyday talk, primarily on the qualities of their process, one 
should judge the deliberative system as a whole on its outcome: 


If a deliberative system works well, it filters out and discards the worst ideas avail- 
able on public matters while it picks up, adopts, and applies the best ideas. If the 
deliberative system works badly, it distorts facts, portrays ideas in forms that their 
originators would disown, and encourages citizens to adopt ways of thinking and 
acting that are good neither for them nor for the larger polity. A deliberative sys- 
tem at its best, like all systems of democratic participation, helps its participants 
understand themselves and their environment better. It also helps them change 
themselves and others in ways that are better for them and better for the whole 
society—though sometimes these goals conflict. 

(Mansbridge 1999, 211-212) 


I concluded that “How one judges a deliberative system thus depends heavily on what 
one believes to be a “good” or “bad” way of thinking or acting and what one judges 
to be a better or worse understanding of self and environment. Such judgments will 
always be heavily contested’ (Ibid.) Because outcomes like these are essentially con- 
testable (Gallie 1955-1956), they are intrinsically resistant to ‘objective’ empirical 
analysis. Even the boundaries of the ‘political’ are ‘explicitly a matter for contest’ 
(Mansbridge 1999, 215). 

Just over two decades later, however, a wide variety of methods for assessing 
deliberative systems have arisen, as captured in this volume. In normative theory, 
subsequent analyses have advanced dimensions for assessing deliberative systems 
other than the epistemic and developmental considerations I argued were so deeply 
contested as to be almost impossible to measure persuasively. Most of these new 
proposed assessment dimensions refer to process, not to outcomes. Dryzek (2009), 
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for example, has argued that we should assess deliberative systems on the degree to 
which they are ‘authentic’ (induce reflection, be non-coercive, and connect partic- 
ular interests to general principles), high on ‘reciprocity’ (make arguments in terms 
others can accept), ‘inclusive’ (involve a large range of interests and discourses), and 
‘consequential’ (have impact on collective decisions or social outcomes). Mansbridge 
et al. (2012) concluded that although deliberative systems could be assessed on many 
contested dimensions, three are relatively uncontroversial: the ‘epistemic’ function 
(inform the public; take relevant considerations from all corners; air, discuss, and 
appropriately weigh those considerations); the ‘ethical’ (promote equal respect); and 
the ‘democratic (include multiple voices, interests, concerns, and claims on the basis 
of feasible equality). All of these are process concerns, although the epistemic func- 
tion bears heavily on outcomes. Niemeyer (2014), going back to first principles, 
argued more generally that deliberative systems should, in the end, be judged on 
the degree to which they promote legitimate decision-making in which ‘citizens get 
to decide, in light of reasons, what should be done’ (citing Chambers 2012 citing 
Cohen 2007; see also Curato and Böker 2016). Dryzek (2017, 630) added that delib- 
erative systems should be ‘judged in terms of the degree to which they facilitate or 
obstruct competent, critical, inclusive, and egalitarian communicative action in the 
development of integrative norms. Empirical scholars may be able to devise approx- 
imate measures for some of these characteristics for assessment, but none is easily 
operationalizable and the measurements may be heavily contested. 

Empirical assessment of specific features of deliberative systems has developed 
imaginatively and rigorously, as this volume indicates. Web-scraping and inductive 
clustering techniques can help greatly with measurement and topic comparison for 
online talk (Franco-Guillén, De Laile, and Parkinson, Chapter 16 in this volume). 
Narrative analysis provides insight into the trajectories of different arguments over 
time and on the broad scale (Boswell, Chapter 23 in this volume). We can also mea- 
sure to some significant degree the deliberative quality of online deliberation (Kies, 
Chapter 10 in this volume). 

Empirical assessment of the deliberative system as a whole is more difficult. Assess- 
ments of ‘macro-deliberatiom (Fleuß, Chapter 9 in this volume) and assessments 
of the deliberative system are not quite the same thing. The assessment of macro- 
deliberation investigates the deliberative quality of important deliberative nodes in 
the system, then aggregates them. It does not include the contributions of non- 
deliberative nodes, such as protest. Social network analysis can identify some of the 
pathologies of deliberative systems.” Discourse analysis based on Q-sort methodol- 
ogy also helps in understanding patterns in the deliberative system (Parry, Chapter 21 
in this volume; Niemeyer and Veri, Chapter 7 in this volume). Yet none of these 
modes of assessment helps measure normative systemic outcomes such as filtering 
out and discarding ‘the worst ideas’ while picking up, adopting, and applying ‘the 
best. 


25 E.g. too tight coupling, as when in some Brazilian councils many members represent civil society and 
public authorities at the same time, or decoupling, when networks have no overlapping connections (da 
Silva, Ribeiro, and Higgins, Chapter 15 in this volume). 


Jane Mansbridge 491 


Nor are these forms of assessment likely to be able to assess the benefits and harms 
to the deliberative system of nodes such as protest. One such problematic node 
is what I called an “enclave” model of democratic deliberation, describing com- 
munities of discourse of the likeminded or those similarly situated within a larger 
deliberative system (Mansbridge 1994, 64, 55). On the negative side, enclave delib- 
eration can reinforce lines of division and enmity between the enclave participants 
and others. On the positive side, it can have, particularly for marginalized groups, 
the functions of promoting clarity in understanding interests and preferences, cre- 
ativity in generating new ideas, and mutual support in getting those ideas before the 
public. Thus, as with protest, even when the discourse in the enclave is far from delib- 
erative, it can contribute positively to the quality of deliberation in the larger system 
(Mansbridge et al. 2012; also Karpowitz, Raphael and Hammond 2009). 

For what we may call the ‘forum-system link; or the connection of citizens’ assem- 
blies and other forums to the larger deliberative system, the norms are still in flux. On 
the broad normative front of what to measure in the forum-system link, Niemeyer 
(2014) argues that deliberative citizens’ forums should have the two functions of what 
he calls ‘deliberation-making’ (in contrast to ‘decision-making’) and ‘capacity build- 
ing within a larger framework of building legitimacy. In ‘deliberation-making, the 
citizens’ forums should distil and synthesize for the public different ways of thinking 
about an issue, acting in the best case as trusted knowledge brokers on the basis of 
the time, expert resources, and deliberative opportunities to which the forum partici- 
pants have privileged access. They might even contribute to a public ‘meta-consensus, 
or larger agreement on the range of legitimate considerations on that issue. In ‘capac- 
ity building, a forum should help the larger system to ‘host inclusive and authentic 
deliberation; in part by acting as exemplars of deliberation. 

On the forum-system link, Curato and Böker argue further for a normative ‘obli- 
gation of seeking legitimacy’ (2016, 177) and a consequent ‘obligation to persuade’ 
the public outside the forum (178), because such forums will succeed in producing 
legitimacy ‘only by convincing the public that their conclusions are valid and their 
recommendations are worth pursuing’ (177-178).”° These formulations of obligation 
raise several questions. 

First, who is obliged to seek legitimacy and persuade —the citizens selected for 
the forum who have agreed to serve, the organizers of the forum, or both? The 
philosopher Wendy Salkin (2021a, 2021b) has plumbed the obligations of citizens 
who become informal representatives of other citizens both when they are ‘witting 
and unwitting’ and when they are ‘willing and unwilling. The randomly chosen par- 
ticipants in citizen deliberative forums occupy a half-way role between informal and 
formal representatives. They are also both ‘conscripts’ and volunteers. Some partic- 
ipants may wittingly and willingly take on the citizen duty to represent others (see 
Jaquet 2019, 648 ff), to seek legitimacy, and to persuade, but some might hesitate even 
to participate in a forum that came with such obligations. Any normative obligation 
to persuade seems thus to fall on the organizers. 


6 See also Beauvais and Warren (2019) for the deliberative forum’s functions of inclusion, communi- 
cation and deliberation, and decision-making in the larger deliberative system. 
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Second, are reasons better than mere trust? Empirical scholars could measure 
probable success in a forum's persuasiveness by comparing Time 1 versus Time 2 
differences in opinion amongst members of the public who have heard about the 
forum’s conclusions versus those who had never heard of the forum. Yet such differ- 
ences might arise simply from trust in the forum participants, not from weighing the 
participants’ distilled reasons (Warren and Gastil 2016; Curato and Boker 2016). Sim- 
ple trust might be cost-efficient for citizens and normatively acceptable. Karpowitz 
and Raphael (2014), however, argue that in the forum-system link, ‘deliberative pub- 
licity’ should focus on the reasons citizens on both sides of an issue give for their 
positions. Studying the formal reports of ten deliberative citizens’ forums, of which 
only one used a randomized selection of citizens, they find that only two, not includ- 
ing the one randomized forum, gave significant attention to reasons. They argue that 
a focus on reasons would contribute more to the deliberative system. 

Third, are reasons better than storytelling? Karpowitz and Raphael also caution 
against forms of deliberative publicity that solely present citizens telling stories about 
their own experiences. A focus only on the ‘authenticity of citizens’ experiential evi- 
dence as expressed in their own voices’ risks one specific and one general harm. 
Specifically, the stories often describe pain and distress, so the practice and reporting 
of storytelling tends to cast marginalized groups in a ‘victim’ role. More generally, 
storytelling de-emphasizes citizens’ capacities to generalize, analyse, and connect 
their experiences with the common good (2014, 262, 264-265). Although reasons 
are not intrinsically better than storytelling, deliberative publicity, they argue, should 
present citizens primarily as reasoning beings. 

Fourth, are reasons given by identifiable people better for legitimacy and persua- 
sion than mere numbers? Maia and Choucair (Chapter 26 in this volume, pp. 381-2) 
point out that in general ‘images and visual elements remain underexplored in studies 
on deliberation’ (see also Mendonga et al. 2020). Equally underexplored are the possi- 
ble positive effects on the public of watching video clips from deliberative forums that 
feature ‘people like me’ giving the reasons they changed their minds on salient issues. 
When it is possible to collect pre- and post-deliberation surveys on a computer-linked 
tablet rather than by pencil-and-paper (or even just to ask all participants on leaving 
if they changed their mind on any issues), it should be easy to identify participants 
who have changed their minds on salient topics and, before they leave the forum’s 
venue, have volunteers interview them with mobile phone cameras on the reasons for 
their changes of mind, using those videos for the subsequent deliberative publicity. 
Admittedly, current research leads one to predict little change amongst disaffected 
members of the public, especially if the results of the citizen forum are contrary to 
the direction the disaffected citizen prefers (Goldberg and Bachtiger 2022). Yet hear- 
ing the reasons for such changes amongst citizens in the forum might have greater 
impact on the public than reading numerical reports of which demographic groups 
were more likely to change, in which direction, and to what degree. And videos of 
particular individuals, especially ‘people like me; explaining their reasons might have 
even greater impact. 
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Fifth, should evaluations be considered part of the persuasive process? Participant 
evaluations designed to bring out criticisms of the deliberative process help practi- 
tioners distinguish believable from tawdry processes and provide guidance for future 
designs. Yet they may also tend to leave participants in a critical frame of mind. 
Evaluations designed to allow participants to keep intact their enthusiasm for the 
process are more likely to enhance the public’s perceptions of the legitimacy of that 
process. In one of the few empirical studies to measure emotion in a deliberative 
forum, self-reported ‘happiness’ peaked on the final day (Johnson et al. 2019). Par- 
ticipant evaluations usually reflect this end-of-process enthusiasm. If the perceived 
legitimacy of these forums rests at all on the later communications of participants 
with other citizens, evaluation questions right at the end that raise doubts and stress 
potential flaws in the process might decrease the forum's legitimacy.” 

Sixth, must deliberative publicity give proportional time and space to the reasons 
both for and against the deliberative forum’s final conclusions? On the one hand, the 
greatest obligation of forums to the deliberative system may be to provide nuance, 
including the range of opposing opinions, in order to invigorate the public debate. On 
the other hand, giving time in deliberative publicity to arguments against the forum's 
final recommendations, especially in proportion to the numbers in the assembly 
holding the opposing opinions, might well undercut the task of convincing the public 
that those recommendations are worth pursuing. In this instance, Niemeyer’s norm 
of ‘deliberation-making’ (give the public both sides of the argument) seems to con- 
flict with Curato and Böker’s norm of ‘legitimacy-seeking’ (persuade the public that 
the forum's conclusions rest on good reasons). 

In the current state of flux regarding the micro-norms of the forum-system link, 
practitioners play a key normative role. Advised of the tensions, they will have to 
make their own best normative judgements on design as they go along. Their experi- 
ence may then give rise to insights to which neither normative theorists nor empirical 
researchers now have access. All the better, then, if practitioners are in ongoing 
recursive contact with normative theorists who have the inclination and intellectual 
contacts to help think these issues through as well as empirical scholars who can help 
operationalize and measure the possible outcomes of both greater deliberative depth 
in the public and greater legitimacy for the forums’ conclusions. 

Process tracing can help us understand patterns of impact in the forum-system 
link (Pickering, Chapter 20 in this volume). But Pickering reminds us that at the 
moment we have little hard evidence that the carefully constructed citizen forums 
dedicated to deliberation enhance debate amongst members of the public. This lack 
of evidence usually arises because of the currently weak causal links between what 
goes on in these forums and the public.’* The disaffected already find reasons to reject 


27 The “peak-end rule” in psychology derives from retrospective pain and pleasure ratings being based 
on a combination of the peaks and final moments of the experience (see summary of evidence in Do, 
Rupert and Wolford 2008). 

28 E.g., Devillers et al. (2021). See also Goodin and Dryzek (2006), cited in Werner and 
Muradova, Chapter 13 in this volume; Elstub et al. (2016), cited in Pickering, Chapter 20 in this 
volume; and Gastil, Chapter 14 in this volume. 
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the conclusions of deliberative citizen forums when they do not like the outcome. If 
such forums tend to lead citizens to favor more liberal outcomes (see Grönlund and 
Herne, Chapter 11 in this volume; and Gastil, Chapter 14 in this volume, p. 210.) 
conservative members of the public may also come to see the forums as suspect.?? 

On the test issue for perceived legitimacy of so many citizens feeling not heard by 
either government or other citizens in their nations, deliberative forums seem not 
to have made much of a dent. Better deliberative publicity would probably help. So 
would elected legislatures, administrative agencies, and NGOs with policy influence 
consulting regularly on specific issues with randomly-selected deliberative citizen 
forums (Mansbridge 2022 on recursive representation). 

Improving the deliberative system in practice is hard. Understanding theoreti- 
cally what norms are appropriate for a system’s improvement and assessment may 
be harder still. Assessing a system empirically may the hardest task of all. 


Conclusion 


The environmental problems we face seem almost insurmountable, the shadow of 
nuclear catastrophe looms, and the ‘free rider’ problems that governments must 
solve, in large part by increasing government coercion, multiply every year. Yet 
human ingenuity has not diminished. On the contrary, new methods of commu- 
nication are linking idealistic, intelligent, and inventive young people around the 
world. When a new technology arises, such as deliberative citizen forums with their 
capacity to strengthen democracy, the young have jumped to implement and adapt 
it. This volume bursts with novel, exciting ideas, peaking in the sections on Exploring 
and Adapting. Practitioners, political theorists, modellers, survey researchers, exper- 
imentalists, qualitative interpretativists, and the mix that is grounded theorists—all 
represented in this volume—together are lending their energy and insight to mak- 
ing deliberative democracy work: for better solutions, better citizens, and greater 
legitimacy. We need it all. 
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